<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Ok I think I have a quick fix for people struggling with jupyter notebook on NFS. <div><br></div><div>There are essentially two parts to the problem. The first one deals with forcing jupyter to create its SQLite db in your scratch directory. The second part is the ipython directory.</div><div><br></div><div><br></div><div>Part 1:</div><div><br></div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Step 1<span> : </span></span>First ssh into the Auton Computing Environment and run</div></div><div><br></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px">$jupyter notebook --generate-config</span><br></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><br></span></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px">This would create a config file 'jupyter_notebook_config.py' in ~/.jupyter . ( Note that ~ this is /zfsauton/home/<username> )</span></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><br></span></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px">Step 2: Edit the file above with your favourite text editor and add (or uncomment) the following line</span></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><br></span></div><div><span style="font-size:14px;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol"">c.NotebookNotary.db_file='/home/scratch/<username>/jupyter.log' </span><br></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><br></span></font></span></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px">replace <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><username> with your auton username. </span></span></font></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">If you executed part 1 correctly, you should be able to create a new Notebook, and stop the Ipython server using control-c. However, you will not be able to connect Jupyter to Ipython for which you need to perform part 2</span></span></font></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Part 2:</span></span></font></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div><div>$export IPYTHONDIR=/home/scratch/<username>/ipython</div></div><div><br></div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px">as always replace<span> </span><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><username> with your auton username. </span></span></font></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">you can also add this to .bashrc for next time ;)</span></span></font></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px">Thats it!</span></font></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><br></span></font></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px">It should work now!</span></font></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><br></span></font></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px">Chirag</span></font></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> </span></span></font></div><br></div><div><br></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div><font color="#24292e" face="-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol"><span style="font-size:14px"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></font></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><br></span></div><div><span style="text-decoration-style:initial;text-decoration-color:initial;box-sizing:border-box;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><br></span></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 6, 2018 at 8:43 PM, Chirag Nagpal <span dir="ltr"><<a href="mailto:chiragn@cs.cmu.edu" target="_blank">chiragn@cs.cmu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I am working on fixing this. <div><br></div><div>It is indeed SQLite and NFS not talking to each other which is the problem. I am able to get jupyter to behave somewhat better by forcing the SQLite server to be in the scratch instead if the NFS. This requires changing some default flags for jupyter. </div><div><br></div><div>Its not completely fixed yet, but I will get back with an update soon. </div><div><br></div><div>Chirag</div><div><br></div><div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Tue, Nov 6, 2018 at 8:39 PM, Predrag Punosevac <span dir="ltr"><<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>Chirag Nagpal <<a href="mailto:chiragn@cs.cmu.edu" target="_blank">chiragn@cs.cmu.edu</a>> wrote:<br>
<br>
> Predrag: I am able to reproduce the error on the scratch directory too.<br>
> <br>
<br>
</span>I am sure you guys have a problem but one doesn't need Jupyter to do<br>
actual programming in Python. I am saying this because there are<br>
handful of you who are affected by this behavior (God knows what could<br>
have caused possibly even regressions by newer version of packages) and<br>
I am the only firefighter currently without bandwidth to deal with such<br>
wild fires.<br>
<span><br>
> Vincent, Matt: are you using the anaconda jupyter? it could be an anaconda<br>
> upgrade thats responsible ?<br>
> <br>
> Chirag<br>
> <br>
> On Tue, Nov 6, 2018 at 4:45 PM, Vincent Jeanselme <<a href="mailto:vjeansel@andrew.cmu.edu" target="_blank">vjeansel@andrew.cmu.edu</a>><br>
> wrote:<br>
> <br>
> > Even after changing my bashrc with export IPYTHONDIR=/home/scratch/$USER<wbr>/.ipython<br>
> > and reinstalling my jupyter, it still does not seem to work. I also have an<br>
> > issue with git, I am no longer able to pull from the server.<br>
> ><br>
<br>
<br>
</span>The git issue is environmental variable issue which is caused by the<br>
fact that there is only one user on Gogs server (git) and all accounts<br>
are just aliases with their own ssh-keys to this account. I don't use<br>
and know enough about git but those nasty files in your<br>
<br>
reponame/.git <br>
<br>
folder <br>
<br>
which look like <br>
<br>
predragp@lov3$ ls<br>
branches description HEAD index logs ORIG_HEAD refs<br>
config FETCH_HEAD hooks info objects packed-refs<br>
<br>
apparently get populated in different ways depending on the login. So<br>
for example if I ssh to one of the computing nodes from home I get this<br>
<br>
<br>
predragp@lov3$ git pull<br>
Host key fingerprint is<br>
SHA256:XEkrneUFkkAPyf0gMOQKa3a<wbr>miAp3QrXmu1x+sIJMcsA<br>
+---[ECDSA 256]---+<br>
| .++++o.o.. |<br>
| . .+=+B . |<br>
|. . . . ..Bo. |<br>
|.E + o . o . |<br>
| .= * S |<br>
|o=o* . |<br>
|+=+.o .o |<br>
|o.o+.+. . |<br>
|. +o.. |<br>
+----[SHA256]-----+<br>
remote: Enumerating objects: 34, done.<br>
remote: Counting objects: 100% (34/34), done.<br>
remote: Compressing objects: 100% (21/21), done.<br>
remote: Total 22 (delta 8), reused 0 (delta 0)<br>
Unpacking objects: 100% (22/22), done.<br>
>From ssh://git:/predragp/ansible<br>
cfc10cf..71a9aec master -> origin/master<br>
Updating cfc10cf..71a9aec<br>
Fast-forward<br>
Linux/autofs/etc/auto.nfs | 18<br>
+++++++++++++++++<br>
Linux/ldap/etc/openldap/{<wbr>certs => cacerts}/ca.crt | 0<br>
Linux/ldap/etc/openldap/ldap.<wbr>conf | 6 +++---<br>
Linux/ldap/etc/sssd/sssd.<wbr>conf | 2 +-<br>
Linux/ldap/ldap.yaml | 24<br>
+++++++++++++++--------<br>
5 files changed, 38 insertions(+), 12 deletions(-)<br>
rename Linux/ldap/etc/openldap/{certs => cacerts}/ca.crt (100%)<br>
<br>
<br>
If I ssh to my desktop I get <br>
<br>
predragp@lake$ git pull<br>
Host key fingerprint is<br>
SHA256:XEkrneUFkkAPyf0gMOQKa3a<wbr>miAp3QrXmu1x+sIJMcsA<br>
+---[ECDSA 256]---+<br>
| .++++o.o.. |<br>
| . .+=+B . |<br>
|. . . . ..Bo. |<br>
|.E + o . o . |<br>
| .= * S |<br>
|o=o* . |<br>
|+=+.o .o |<br>
|o.o+.+. . |<br>
|. +o.. |<br>
+----[SHA256]-----+<br>
Password for <a href="mailto:git@git.int.autonlab.org" target="_blank">git@git.int.autonlab.org</a>:<br>
<br>
which is the indication that my .ssh/config file and the ssh-key were<br>
not read even though<br>
<br>
Host git <br>
HostName <a href="http://git.int.autonlab.org" rel="noreferrer" target="_blank">git.int.autonlab.org</a><br>
Port 2222<br>
User git <br>
IdentityFile /home/predragp/.ssh/git_rsa <br>
<br>
However if I log from the terminal to my desktop I don't have a Git<br>
issue. <br>
<br>
<br>
Best,<br>
Predrag<br>
<div><div><br>
<br>
<br>
<br>
<br>
<br>
> ><br>
> > On 11/6/18 3:14 PM, Predrag Punosevac wrote:<br>
> ><br>
> >> Matthew Barnes <<a href="mailto:mbarnes1@andrew.cmu.edu" target="_blank">mbarnes1@andrew.cmu.edu</a>> wrote:<br>
> >><br>
> >> Also having this problem. Trying to create a new notebook hangs on<br>
> >>> "Creating new notebook in", and unable to open old notebooks. Anyone's<br>
> >>> setup currently working?<br>
> >>><br>
> >>> Jupyter Notebook is using sqlite database to store the info. Unless you<br>
> >> explicitly force Jupyter to create the database on the scratch directory<br>
> >> the database is stored on the NFS share. There is nothing worse one can<br>
> >> do in terms of data consistency than put a database or a private Git<br>
> >> repo (talking about the server) onto the NFS. The datebase was left in<br>
> >> inconsistent state after the file server was rebooted. You have to clear<br>
> >> it and possibly recreate the database to be able to use Jupyter<br>
> >> Notebook.<br>
> >><br>
> >> Best,<br>
> >> Predrag<br>
> >><br>
> >><br>
> >> On Tue, Nov 6, 2018 at 12:44 PM Vincent Jeanselme <<br>
> >>> <a href="mailto:vjeansel@andrew.cmu.edu" target="_blank">vjeansel@andrew.cmu.edu</a>><br>
> >>> wrote:<br>
> >>><br>
> >>> Hello,<br>
> >>>><br>
> >>>> Has anyone faced problems with running Jupyter Notebook since yesterday<br>
> >>>> ?<br>
> >>>> Did you remember what was the change to operate after the last reboot of<br>
> >>>> the sqlite database ?<br>
> >>>><br>
> >>>> Thank you,<br>
> >>>><br>
> >>>> Vincent<br>
> >>>><br>
> >>>> --<br>
> >>>> Vincent Jeanselme<br>
> >>>> -----------------<br>
> >>>> Analyst Researcher<br>
> >>>> Auton Lab - Robotics Institute<br>
> >>>> Carnegie Mellon University<br>
> >>>><br>
> >>>><br>
> >>>> --<br>
> > Vincent Jeanselme<br>
> > -----------------<br>
> > Analyst Researcher<br>
> > Auton Lab - Robotics Institute<br>
> > Carnegie Mellon University<br>
> ><br>
> ><br>
> <br>
> <br>
> -- <br>
> <br>
</div></div>> *Chirag Nagpal* Graduate Student, Language Technologies Institute<br>
<div><div>> School of Computer Science<br>
> Carnegie Mellon University<br>
> <a href="http://cs.cmu.edu/~chiragn" rel="noreferrer" target="_blank">cs.cmu.edu/~chiragn</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span><b>Chirag Nagpal<br></b><span class="">
Graduate Student, Language Technologies Institute<br>
School of Computer Science<br>
Carnegie Mellon University<br></span></span></span></span></div><div><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span><a href="http://cs.cmu.edu/~chiragn" target="_blank">cs.cmu.edu/~chiragn</a><br></span></span></span></div></div></div></div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span><b>Chirag Nagpal<br></b>
Graduate Student, Language Technologies Institute<br>
School of Computer Science<br>
Carnegie Mellon University<br></span></span></span></div><div><span style="color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span><a href="http://cs.cmu.edu/~chiragn" target="_blank">cs.cmu.edu/~chiragn</a><br></span></span></span></div></div></div></div></div></div>
</div>