Jupyter Notebooks
Chirag Nagpal
chiragn at cs.cmu.edu
Tue Nov 6 21:39:30 EST 2018
Ok I think I have a quick fix for people struggling with jupyter notebook
on NFS.
There are essentially two parts to the problem. The first one deals with
forcing jupyter to create its SQLite db in your scratch directory. The
second part is the ipython directory.
Part 1:
Step 1 : First ssh into the Auton Computing Environment and run
$jupyter notebook --generate-config
This would create a config file 'jupyter_notebook_config.py' in ~/.jupyter
. ( Note that ~ this is /zfsauton/home/<username> )
Step 2: Edit the file above with your favourite text editor and add (or
uncomment) the following line
c.NotebookNotary.db_file='/home/scratch/<username>/jupyter.log'
replace <username> with your auton username.
If you executed part 1 correctly, you should be able to create a new
Notebook, and stop the Ipython server using control-c. However, you will
not be able to connect Jupyter to Ipython for which you need to perform
part 2
Part 2:
$export IPYTHONDIR=/home/scratch/<username>/ipython
as always replace <username> with your auton username.
you can also add this to .bashrc for next time ;)
Thats it!
It should work now!
Chirag
On Tue, Nov 6, 2018 at 8:43 PM, Chirag Nagpal <chiragn at cs.cmu.edu> wrote:
> I am working on fixing this.
>
> It is indeed SQLite and NFS not talking to each other which is the
> problem. I am able to get jupyter to behave somewhat better by forcing the
> SQLite server to be in the scratch instead if the NFS. This requires
> changing some default flags for jupyter.
>
> Its not completely fixed yet, but I will get back with an update soon.
>
> Chirag
>
>
>
> On Tue, Nov 6, 2018 at 8:39 PM, Predrag Punosevac <predragp at andrew.cmu.edu
> > wrote:
>
>> Chirag Nagpal <chiragn at cs.cmu.edu> wrote:
>>
>> > Predrag: I am able to reproduce the error on the scratch directory too.
>> >
>>
>> I am sure you guys have a problem but one doesn't need Jupyter to do
>> actual programming in Python. I am saying this because there are
>> handful of you who are affected by this behavior (God knows what could
>> have caused possibly even regressions by newer version of packages) and
>> I am the only firefighter currently without bandwidth to deal with such
>> wild fires.
>>
>> > Vincent, Matt: are you using the anaconda jupyter? it could be an
>> anaconda
>> > upgrade thats responsible ?
>> >
>> > Chirag
>> >
>> > On Tue, Nov 6, 2018 at 4:45 PM, Vincent Jeanselme <
>> vjeansel at andrew.cmu.edu>
>> > wrote:
>> >
>> > > Even after changing my bashrc with export
>> IPYTHONDIR=/home/scratch/$USER/.ipython
>> > > and reinstalling my jupyter, it still does not seem to work. I also
>> have an
>> > > issue with git, I am no longer able to pull from the server.
>> > >
>>
>>
>> The git issue is environmental variable issue which is caused by the
>> fact that there is only one user on Gogs server (git) and all accounts
>> are just aliases with their own ssh-keys to this account. I don't use
>> and know enough about git but those nasty files in your
>>
>> reponame/.git
>>
>> folder
>>
>> which look like
>>
>> predragp at lov3$ ls
>> branches description HEAD index logs ORIG_HEAD refs
>> config FETCH_HEAD hooks info objects packed-refs
>>
>> apparently get populated in different ways depending on the login. So
>> for example if I ssh to one of the computing nodes from home I get this
>>
>>
>> predragp at lov3$ git pull
>> Host key fingerprint is
>> SHA256:XEkrneUFkkAPyf0gMOQKa3amiAp3QrXmu1x+sIJMcsA
>> +---[ECDSA 256]---+
>> | .++++o.o.. |
>> | . .+=+B . |
>> |. . . . ..Bo. |
>> |.E + o . o . |
>> | .= * S |
>> |o=o* . |
>> |+=+.o .o |
>> |o.o+.+. . |
>> |. +o.. |
>> +----[SHA256]-----+
>> remote: Enumerating objects: 34, done.
>> remote: Counting objects: 100% (34/34), done.
>> remote: Compressing objects: 100% (21/21), done.
>> remote: Total 22 (delta 8), reused 0 (delta 0)
>> Unpacking objects: 100% (22/22), done.
>> From ssh://git:/predragp/ansible
>> cfc10cf..71a9aec master -> origin/master
>> Updating cfc10cf..71a9aec
>> Fast-forward
>> Linux/autofs/etc/auto.nfs | 18
>> +++++++++++++++++
>> Linux/ldap/etc/openldap/{certs => cacerts}/ca.crt | 0
>> Linux/ldap/etc/openldap/ldap.conf | 6 +++---
>> Linux/ldap/etc/sssd/sssd.conf | 2 +-
>> Linux/ldap/ldap.yaml | 24
>> +++++++++++++++--------
>> 5 files changed, 38 insertions(+), 12 deletions(-)
>> rename Linux/ldap/etc/openldap/{certs => cacerts}/ca.crt (100%)
>>
>>
>> If I ssh to my desktop I get
>>
>> predragp at lake$ git pull
>> Host key fingerprint is
>> SHA256:XEkrneUFkkAPyf0gMOQKa3amiAp3QrXmu1x+sIJMcsA
>> +---[ECDSA 256]---+
>> | .++++o.o.. |
>> | . .+=+B . |
>> |. . . . ..Bo. |
>> |.E + o . o . |
>> | .= * S |
>> |o=o* . |
>> |+=+.o .o |
>> |o.o+.+. . |
>> |. +o.. |
>> +----[SHA256]-----+
>> Password for git at git.int.autonlab.org:
>>
>> which is the indication that my .ssh/config file and the ssh-key were
>> not read even though
>>
>> Host git
>> HostName git.int.autonlab.org
>> Port 2222
>> User git
>> IdentityFile /home/predragp/.ssh/git_rsa
>>
>> However if I log from the terminal to my desktop I don't have a Git
>> issue.
>>
>>
>> Best,
>> Predrag
>>
>>
>>
>>
>>
>>
>> > >
>> > > On 11/6/18 3:14 PM, Predrag Punosevac wrote:
>> > >
>> > >> Matthew Barnes <mbarnes1 at andrew.cmu.edu> wrote:
>> > >>
>> > >> Also having this problem. Trying to create a new notebook hangs on
>> > >>> "Creating new notebook in", and unable to open old notebooks.
>> Anyone's
>> > >>> setup currently working?
>> > >>>
>> > >>> Jupyter Notebook is using sqlite database to store the info. Unless
>> you
>> > >> explicitly force Jupyter to create the database on the scratch
>> directory
>> > >> the database is stored on the NFS share. There is nothing worse one
>> can
>> > >> do in terms of data consistency than put a database or a private Git
>> > >> repo (talking about the server) onto the NFS. The datebase was left
>> in
>> > >> inconsistent state after the file server was rebooted. You have to
>> clear
>> > >> it and possibly recreate the database to be able to use Jupyter
>> > >> Notebook.
>> > >>
>> > >> Best,
>> > >> Predrag
>> > >>
>> > >>
>> > >> On Tue, Nov 6, 2018 at 12:44 PM Vincent Jeanselme <
>> > >>> vjeansel at andrew.cmu.edu>
>> > >>> wrote:
>> > >>>
>> > >>> Hello,
>> > >>>>
>> > >>>> Has anyone faced problems with running Jupyter Notebook since
>> yesterday
>> > >>>> ?
>> > >>>> Did you remember what was the change to operate after the last
>> reboot of
>> > >>>> the sqlite database ?
>> > >>>>
>> > >>>> Thank you,
>> > >>>>
>> > >>>> Vincent
>> > >>>>
>> > >>>> --
>> > >>>> Vincent Jeanselme
>> > >>>> -----------------
>> > >>>> Analyst Researcher
>> > >>>> Auton Lab - Robotics Institute
>> > >>>> Carnegie Mellon University
>> > >>>>
>> > >>>>
>> > >>>> --
>> > > Vincent Jeanselme
>> > > -----------------
>> > > Analyst Researcher
>> > > Auton Lab - Robotics Institute
>> > > Carnegie Mellon University
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > *Chirag Nagpal* Graduate Student, Language Technologies Institute
>> > School of Computer Science
>> > Carnegie Mellon University
>> > cs.cmu.edu/~chiragn
>>
>
>
>
> --
>
> *Chirag Nagpal* Graduate Student, Language Technologies Institute
> School of Computer Science
> Carnegie Mellon University
> cs.cmu.edu/~chiragn
>
--
*Chirag Nagpal* Graduate Student, Language Technologies Institute
School of Computer Science
Carnegie Mellon University
cs.cmu.edu/~chiragn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20181106/329d1723/attachment-0001.html>
More information about the Autonlab-users
mailing list