Data Management and Backup
GM/CA @ APS runs 10Gb network and users are welcome to SFTP out
their data to the servers at their home institutions. External access to the
GM/CA SFTP and Globus GridFTP
servers is also possible, but needs to be requested in advance. The access can
be allocated for the time of experiment plus one day. That policy is enforced
in order to avoid conflict of resources between different user groups. The real
rate of remote data transfer will depend on the network connectivity to the users
institution. Please check the Network Speed
Testingb> page. Unless there is a bottleneck on user's side, one can expect
the SFTP rates of 5-8MB/s and Globus GridFTP
rates of 20-25MB/s. For a 200GB data set it may take about 10 and 3 hours
Please be adviced to never using Wi-Fi connnections
for network data transfers as it drastically reduces speeds, especially at long
GM/CA @ APS recommends and supports external USB
disks as inexpensive and fast way to backup users data. We strongly recommend
using USB3 drives due to much higher speeds compared to USB2.
The table below provides a speed comparison of different connector types for
typical 5400rpm HDD external drives (the advantages of USB3 for SSD drives are
even higher as the speeds may reach 200-300 MB/s):
| Connector || Bus Speed || Real Speed || 200GB backup time || Restricted by |
| USB3 || 500 MB/s || 40-50 MB/s || 1 hour || disk spindle |
| USB2 || 40 MB/s || 20 MB/s || 3 hours || bus |
Not all computers may have USB-3 ports. Please ask your host if in trouble finding
the right hub.
Some users have problems distinguishing between the USB-3 and USB-2
ports. Please look at the labeling and the color. The USB-3 ports typically have
blue connectors and cables and/or the SS (Super-Speed) marking.
Support for DVD disks is not provided because at the rate of 200GB of data
per experiment one would have to burn more than 45 disks.
Supported disk formats
The supported formats on external disks are FAT32,
NTFS, exFAT, ext2/ext3/ext4,
xfs, and HFS+.
We have only Linux computers at the beamlines. Therefore some Windows-specific
and MacOSX-specific extensions, for example encrypted drives like WD Passport
that require some Windows or MacOSX utility to unlock the encrypted partition,
may not work. When choosing format for your hard drive, keep in mind that FAT32
has the best record in terms of portability between different operating systems
(Windows, Linux and MacOSX), while HFS+ has the shortest history of support
under Linux. Please, try to avoid bringing disks with multiple partitions and
disks with already stored data as in some problematic cases your disk may need
to be re-partitioned or reformatted.
The HFS+ format is supported on the second-day
workstations ws4, ws5, and ws7 only where we install a commercial HFS+ driver
Please consult this HFS+ manual for the details.
USB disks can be connected to any beamline workstation except for ws2 and
ws3 as they may be controlling the experiment . In the second-day area USB-3 disks can
be connected to most workstations with the SS port label (see the marking examples
above). Usually disks are mounted automatically upon connecting provided user has logged
in before connecting the disk. If the disk is not auto-mounted, go to Applications
-> Accessories > Disks, then find your disk in the
left-pane list, choose the right partition and click the "Mount
Selected Partition" button:
Additional details of mounting external disks under Linux operating system
are described elsewhere.
For the latest changes with the backup procedures and detailed technical
instructions please refer the Backup Manual