GPU2 Fwd: Autonlab-sysinfo Digest, Vol 52, Issue 12

Sat Nov 3 19:07:37 EDT 2018

Dear Autonians,

GPU2 server is officially broke. Please see below. What you are not
going to see in the below report is that besides /root being full due to
the accessive cashing in the /tmp folder which resides on the root
partition /home/scratch is also full.

I have emailed multiple times asking people to clear the /scratch
directory but no aval. I will have to use more heavy handed methods now
like periodically deleteing all /home/scratch directories on all GPU
machines so that machines are usable again.

Best,
Predrag

P.S. I am going for a stroll. If /home/scratch and cash is not clear I
will have to reboot the machine and delete /home/scratch around 9:00 PM
tonight.

-------- Original Message --------
From: autonlab-sysinfo-request at autonlab.org
Subject: Autonlab-sysinfo Digest, Vol 52, Issue 12
To: autonlab-sysinfo at autonlab.org
Date: Sat, 03 Nov 2018 17:21:15 -0400

Send Autonlab-sysinfo mailing list submissions to
	autonlab-sysinfo at autonlab.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://mailman.srv.cs.cmu.edu/mailman/listinfo/autonlab-sysinfo
or, via email, send a message with subject or body 'help' to
	autonlab-sysinfo-request at autonlab.org

You can reach the person managing the list at
	autonlab-sysinfo-owner at autonlab.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Autonlab-sysinfo digest..."

Today's Topics:

   1. Cron <root at gpu2> /usr/lib64/sa/sa1 1 1 ((Cron Daemon))
   2. Cron <root at gpu2> run-parts /etc/cron.hourly ((Cron Daemon))
   3. Cron <root at gpu2> /usr/lib64/sa/sa1 1 1 ((Cron Daemon))
   4. Cron <root at gpu2> /usr/lib64/sa/sa1 1 1 ((Cron Daemon))
   5. Cron <root at gpu2> /usr/lib64/sa/sa1 1 1 ((Cron Daemon))

----------------------------------------------------------------------

Message: 1
Date: Sat,  3 Nov 2018 15:50:01 -0400 (EDT)
From: "(Cron Daemon)" <auton.sysnotify at gmail.com>
To: root at gpu2.int.autonlab.org
Subject: Cron <root at gpu2> /usr/lib64/sa/sa1 1 1
Message-ID: <20181103212106.67CDC1596247 at gpu2.int.autonlab.org>
Content-Type: text/plain; charset=UTF-8

Cannot write data to system activity file: No space left on device

------------------------------

Message: 2
Date: Sat,  3 Nov 2018 16:10:45 -0400 (EDT)
From: "(Cron Daemon)" <auton.sysnotify at gmail.com>
To: root at gpu2.int.autonlab.org
Subject: Cron <root at gpu2> run-parts /etc/cron.hourly
Message-ID: <20181103212106.8CFB71596243 at gpu2.int.autonlab.org>
Content-Type: text/plain; charset=UTF-8

/etc/cron.hourly/0yum-hourly.cron:

Traceback (most recent call last):
  File "/usr/sbin/yum-cron", line 729, in <module>
    main()
  File "/usr/sbin/yum-cron", line 726, in main
    base.updatesCheck()
  File "/usr/sbin/yum-cron", line 618, in updatesCheck
    self.populateUpdateMetadata()
  File "/usr/sbin/yum-cron", line 422, in populateUpdateMetadata
    self.pkgSack # honor skip_if_unavailable
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1074, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 778, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 347, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
    dl = repo._async and repo._commonLoadRepoXML(repo)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1477, in _commonLoadRepoXML
    if self._latestRepoXML(local):
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1446, in _latestRepoXML
    oxml = self._saveOldRepoXML(local)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1303, in _saveOldRepoXML
    shutil.copy2(local, old_local)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 28] No space left on device: '/var/cache/yum/x86_64/7/SCL-core/repomd.xml.old.tmp'
/etc/cron.hourly/ghc-doc-index:

/usr/bin/ghc-doc-index: line 27: /var/lib/ghc/pkg-dir.cache.new: No space left on device
diff: /var/lib/ghc/pkg-dir.cache.new: No such file or directory
haddock: internal error: .: copyFile: resource exhausted (No space left on device)

------------------------------

Message: 3
Date: Sat,  3 Nov 2018 16:00:01 -0400 (EDT)
From: "(Cron Daemon)" <auton.sysnotify at gmail.com>
To: root at gpu2.int.autonlab.org
Subject: Cron <root at gpu2> /usr/lib64/sa/sa1 1 1
Message-ID: <20181103212106.7C6CA1596246 at gpu2.int.autonlab.org>
Content-Type: text/plain; charset=UTF-8

Cannot write data to system activity file: No space left on device

------------------------------

Message: 4
Date: Sat,  3 Nov 2018 15:40:01 -0400 (EDT)
From: "(Cron Daemon)" <auton.sysnotify at gmail.com>
To: root at gpu2.int.autonlab.org
Subject: Cron <root at gpu2> /usr/lib64/sa/sa1 1 1
Message-ID: <20181103212106.8282C1596247 at gpu2.int.autonlab.org>
Content-Type: text/plain; charset=UTF-8

Cannot write data to system activity file: No space left on device

------------------------------

Message: 5
Date: Sat,  3 Nov 2018 15:20:01 -0400 (EDT)
From: "(Cron Daemon)" <auton.sysnotify at gmail.com>
To: root at gpu2.int.autonlab.org
Subject: Cron <root at gpu2> /usr/lib64/sa/sa1 1 1
Message-ID: <20181103212106.742761596243 at gpu2.int.autonlab.org>
Content-Type: text/plain; charset=UTF-8

Cannot write data to system activity file: No space left on device

------------------------------

Subject: Digest Footer

_______________________________________________
Autonlab-sysinfo mailing list
Autonlab-sysinfo at autonlab.org
https://mailman.srv.cs.cmu.edu/mailman/listinfo/autonlab-sysinfo

------------------------------

End of Autonlab-sysinfo Digest, Vol 52, Issue 12
************************************************