git-annex-4.20130501.1: debian/git-annex/usr/share/doc/git-annex/html/design/assistant/cloud.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>cloud</title>
<link rel="stylesheet" href="../../style.css" type="text/css" />
<link rel="stylesheet" href="../../local.css" type="text/css" />
</head>
<body>
<div class="page">
<div class="pageheader">
<div class="header">
<span>
<span class="parentlinks">
<a href="../../index.html">git-annex</a>/
<a href="../../design.html">design</a>/
<a href="../assistant.html">assistant</a>/
</span>
<span class="title">
cloud
</span>
</span>
</div>
</div>
<div class="sidebar">
<p><img src="../../logo_small.png" width="150" height="115" class="img" /></p>
<ul>
<li><a href="../../install.html">install</a></li>
<li><a href="../../assistant.html">assistant</a></li>
<li><a href="../../walkthrough.html">walkthrough</a></li>
<li><a href="../../tips.html">tips</a></li>
<li><span class="createlink">bugs</span></li>
<li><span class="createlink">todo</span></li>
<li><span class="createlink">forum</span></li>
<li><a href="../../comments.html">comments</a></li>
<li><a href="../../contact.html">contact</a></li>
<li><a href="http://flattr.com/thing/84843/git-annex"><img src="https://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" /></a></li>
</ul>
</div>
<div id="pagebody">
<div id="content">
<p>The <a href="./syncing.html">syncing</a> design assumes the network is connected. But it's often
not in these pre-IPV6 days, so the cloud needs to be used to bridge between
LANS.</p>
<h2>The cloud notification problem (<strong>done</strong>)</h2>
<p>Alice and Bob have repos, and there is a cloud remote they both share.
Alice adds a file; the assistant transfers it to the cloud remote.
How does Bob find out about it?</p>
<p>There are two parts to this problem. Bob needs to find out that there's
been a change to Alice's git repo. Then he needs to pull from Alice's git repo,
or some other repo in the cloud she pushed to. Once both steps are done,
the assistant will transfer the file from the cloud to Bob.</p>
<ul>
<li>dvcs-autosync uses xmppp; all repos need to have the same xmpp account
configured, and send self-messages. An alternative would be to have
different accounts that join a channel or message each other. Still needs
account configuration.</li>
<li>irc could be used. With a default irc network, and an agreed-upon channel,
no configuration should be needed. IRC might be harder to get through
some firewalls, and is prone to netsplits, etc. IRC networks have reasons
to be wary of bots using them. Only basic notifications could be done over
irc, as it has little security.</li>
<li>When there's a ssh server involved, code could be run on it to notify
logged-in clients. But this is not a general solution to this problem.</li>
<li>pubsubhubbub does not seem like an option; its hubs want to pull down
a feed over http.</li>
</ul>
<p>See <a href="./xmpp.html">xmpp</a> for design of git-annex's use of xmpp for push notifications.</p>
<h2>storing git repos in the cloud <strong>done for XMPP</strong></h2>
<p>Of course, one option is to just use github etc to store the git repo.</p>
<p>Two things can store git repos in Amazon S3:
* <a href="http://gabrito.com/post/storing-git-repositories-in-amazon-s3-for-high-availability">http://gabrito.com/post/storing-git-repositories-in-amazon-s3-for-high-availability</a>
* <a href="http://wiki.cs.pdx.edu/oss2009/index/projects/gits3.html">http://wiki.cs.pdx.edu/oss2009/index/projects/gits3.html</a></p>
<p>Another option is to not store the git repo in the cloud, but push/pull
peer-to-peer. When peers cannot directly talk to one-another, this could be
bounced through something like XMPP. This is <strong>done</strong> for <a href="./xmpp.html">xmpp</a>!</p>
<p>Another option: Use <a href="https://github.com/blake2-ppc/git-remote-gcrypt">https://github.com/blake2-ppc/git-remote-gcrypt</a> to store
git repo encrypted on cloud storage.</p>
</div>
<div id="comments">
<div class="feedlink">
</div>
<div class="comment" id="comment-bff38388465cb7c83fd1440853a45410">
<div class="comment-subject">
<a href="/design/assistant/cloud.html#comment-bff38388465cb7c83fd1440853a45410">is ftp an option?</a>
</div>
<div class="inlinecontent">
for people only having ftp-access to there storage.
</div>
<div class="comment-header">
Comment by
<span class="author" title="OpenID">
<a href="https://www.google.com/accounts/o8/id?id=AItOawn7Oyqusvn0oONFtVhCx5gRAcvPjyRMcBI">Michaël</a>
</span>
— <span class="date">Wed May 30 06:44:12 2012</span>
</div>
<div style="clear: both"></div>
</div>
<div class="comment" id="comment-5aea3eb7e8aec1e2110b6a4f1a843433">
<div class="comment-subject">
<a href="/design/assistant/cloud.html#comment-5aea3eb7e8aec1e2110b6a4f1a843433">Cloud Service Limitations</a>
</div>
<div class="inlinecontent">
<p>Hey Joey!</p>
<p>I'm not very tech savvy, but here is my question.
I think for all cloud service providers, there is an upload limitation on how big one file may be.
For example, I can't upload a file bigger than 100 MB on box.net.
Does this affect git-annex at all? Will git-annex automatically split the file depending on the cloud provider or will I have to create small RAR archives of one large file to upload them?</p>
<p>Thanks!
James</p>
</div>
<div class="comment-header">
Comment by
<span class="author" title="OpenID">
<a href="https://www.google.com/accounts/o8/id?id=AItOawkq0-zRhubO6kR9f85-5kALszIzxIokTUw">James</a>
</span>
— <span class="date">Sun Jun 10 22:15:04 2012</span>
</div>
<div style="clear: both"></div>
</div>
<div class="comment" id="comment-c0b4e1ebdb4ac122b1b1e8723336544e">
<div class="comment-subject">
<a href="/design/assistant/cloud.html#comment-c0b4e1ebdb4ac122b1b1e8723336544e">re: cloud</a>
</div>
<div class="inlinecontent">
Yes, git-annex has to split files for certian providers. I already added support for this as part of my first pass at supporting box.com, see <a href="../../tips/using_box.com_as_a_special_remote.html">using box.com as a special remote</a>.
</div>
<div class="comment-header">
Comment by
<span class="author" title="OpenID">
<a href="http://joeyh.name/">joeyh.name</a>
</span>
— <span class="date">Mon Jun 11 00:48:08 2012</span>
</div>
<div style="clear: both"></div>
</div>
<div class="comment" id="comment-5e27fcb1377b57315d90fc31268e4abb">
<div class="comment-subject">
<a href="/design/assistant/cloud.html#comment-5e27fcb1377b57315d90fc31268e4abb">OwnCloud</a>
</div>
<div class="inlinecontent">
<p>“Google drive (attractive because it's free, only 5 gb tho)”</p>
<p>Just in case somebody also wants their 5GB of disk space in the cloud, consider using some of the <a href="http://owncloud.org/providers/">owncloud providers</a>. They also offer that amount, but they use free software for everything, using standard protocols (WebDav mostly, because is well supported in all OS).</p>
<p>Git Annex works with them through davfs2, but it would be great if it could support this other program/protocol (OwnCloud/WebDAV) in a more integrated way.</p>
</div>
<div class="comment-header">
Comment by
<span class="author" title="OpenID">
<a href="https://www.google.com/accounts/o8/id?id=AItOawk9XEh8pxrJxZxIkyK7lWaA7QG1UWt9lgU">Gugelplus</a>
</span>
— <span class="date">Mon Aug 27 16:43:19 2012</span>
</div>
<div style="clear: both"></div>
</div>
<div class="addcomment">Comments on this page are closed.</div>
</div>
</div>
<div id="footer" class="pagefooter">
<div id="pageinfo">
<div id="backlinks">
Links:
<a href="./comment_15_68c98a27083567f20c2e6bc2a760991b.html">comment 15 68c98a27083567f20c2e6bc2a760991b</a>
<a href="./polls/prioritizing_special_remotes.html">polls/prioritizing special remotes</a>
<a href="./xmpp.html">xmpp</a>
</div>
<div class="pagedate">
Last edited <span class="date">Tue Jan 29 17:30:51 2013</span>
<!-- Created <span class="date">Tue Jan 29 17:30:51 2013</span> -->
</div>
</div>
<!-- from git-annex -->
</div>
</div>
</body>
</html>