FTS users: LHCb experiment

GFAL / LCG_Util 1.11.16 release

There has been no blog post for almost half a year. It does not mean that nothing has happened since than. We devoted enormous effort to some background works (automated test bed, nightly builds and test runs, change to EMI era from EGEE, etc.). We will test the tools and the procedures in the first months of 2011, analyze if they have added value and how they could be improved. As for the visible part, we released GFAL/LCG_Util 1.11.16 (finally) in November - see the release notes. Better later than never!

FTS 2.2.4 released

Actually, the moment when an FTS version is released is a bit obscure, because we release/certify independently to different platforms, gLite releases, etc. So, in this blog, I will announce "released" when:

- It has been certified / verified for at least one of the supported platforms
- The YUM repository is prepared, so users can install it.

FTS 2.2.4 has fulfilled the above criterion, so it is released :) You can find the release notes here:

FTS 2.2.4 release notes

The release notes have a new format: we use SVN Trac to publish the notes for the future (instead of TWiki), and the release notes will contain a lot more information on one single page than a list of Savannah bug titles.

FTS yaim reference

The FTS Yaim reference pages have been updated:

https://svnweb.cern.ch/trac/glitefts/wiki/FTSYaimReference

Report us please if you find any inconsistencies!

FTS at MAGIC collaboration

Another FTS user: the MAGIC (Major Atmospheric Gamma-ray Imaging Cherenkov Telescope) collaboration. The MAGIC-II experiment consists of a system of two Imaging Cherenkov Telescopes located on the Canary Island La Palma, Spain. They have been designed to study the universe and discover new Gamma-ray sources in the energy range from 50 GeV to 5 TeV. These telescopes have a 17 m diameter reflector, being the largest Cherenkov telescopes in the world, and will be operated in stereoscopic mode for an enhanced sensitivity.

The MAGIC data center hosted by the Port d'Informaci— Cient’fica (PIC) in Barcelona is migrating its services to Grid as part of an upgrade needed to deal with the increased data volume. After migrating the data to a Grid filesystem, they have ported FTS. In the last months they set up an SRM endpoint in the observation site and ported all the data transfer tools to use Grid file transfers.

http://magic.pic.es

FTS 2.2.3 deployment completed

On 25 March, all the Tier1 sites reported successful deployment of FTS 2.2.3. CERN (Tier0) also upgraded the service. The official LHC scientific program will start on 30 March, with 7 TeV collisions, so we were able to deliver and deploy in time. Mission completed :)

FTS at Belle experiment (KEK, Japan)

Nice to know that the world beyond CERN and the LHC Computing Grid also uses FTS. An example is the Belle experiment at the National Laboratory for High Energy Physics (KEK), in Japan. Belle studies the origin of CP violation phenomena, and their results lead to the Nobel Prize for Makoto Kobayashi and Toshihide Maskawa, in 2008.

Do you know about other applications of FTS, GFAL and LCG_Util? We would like to collect them, so in this case let us know please :)

Vnode works

How to work with virtual nodes, FTS, GFAL at CERN? A guide for developers can be found here.

FTS 2.2.2: rolling out

FTS version 2.2.2 (SL4) proceeded to "rolling out" phase. The main updates:

- By default, SRM/gridFTP actions are no longer split. One can re-enable the FTS 2.2.0 behaviour by setting the

FTA_TYPEDEFAULT_URLCOPY_AGENT_SRM_GRIDFTP_SPLIT

Yaim configuration option. See the related Savannah task.

- More liberal checksum handling: algorithm names should follow the specifications, however, some endpoints do not follow them yet. Temporarily, we enabled using their conventions with warnings in the logs. See the related Savannah task.

- Some relax on requirements on how long an SRM TURL is kept valid.

SVN migration + nightlies

GFAL and LCG_Util has been migrated to SVN, no more write access in CVS allowed. The last version that is available from CVS is 1.11.14. FTS migration is ongoing.

To improve the development process, we have finally set up the nightly build system. The first supported platforms are sl5_x86_64_gcc412 and deb5_x86_64_gcc432. The builds are executed at 1:00am (UTC+1) every night, they build the HEAD configuration in ETICS.

Reports are here.

FTS is coming soon. The current status of the nightly build system is accessible here:

https://twiki.cern.ch/twiki/bin/view/EGEE/DMNightlyBuilds

Thanks for everything, Ákos!

Ákos has left the LHC Computing Grid projects. He has coordinated the development and support of the EGEE/LCG grid data management systems, and lead the FTS project since 2002. As he was always short, straight and to-the-point, following his style, I simply should not write more than

that :)

Your comments here are always welcome!

FTS 2.2.1: rolling out

FTS version 2.2.1 proceeded to "rolling out" phase. This version contains the finalization of the checksum support. The database schema has been changed, one has to follow the instructions of the Yaim script to do the upgrade. The API between the transfer-agents and transfer-url-copy has changed, so one has to stop and drain the channels before doing an upgrade.

FTS 2.2.3 certified

Today, the FTS 2.2.3 version has been certified. The pilot service has been updated with the new version. There have been no problems reported so far. The following changes are included:

FTS: agent crash when picking up a job

Couldn't set the private key

They are the infamous "agent crash" and "proxy delegation" problems that prevented from FTS 2.2 production deployment.

The old delegation race condition is back

As it turns out the old delegation race condition came back hunting us again in FTS 2.2, even thought the fix was released to production almost a year ago.

There is a new 'hand built' glite-data-tranfsfer-fts v3.7.0-3 RPM to provide a quick fix for FTS 2.2 as well and hopefully the proper glite-security-delegation-java v1.6.0 will be also included in the gLite build configuration that future releases will not have this problem again.

As wrapping up the year we have a GFAL/lcg_util v1.11.13, FTS 2.2.4 and DPM/LFC v1.7.4.

Details are in the data management relese notes.

This FTS snapshot already has a secure preview of the administrative web interface:

GFAL/lcg_util in SVN

GFAL and lcg_util is the first candidate for migration to SVN.

The new repository is available and I have managed to run remote ETICS builds on all platforms using the glite-data-dm-util_R_1_11_12_3 and glite-data-gfal_R_1_11_12_3 configurations.

While SVN feels much better while renaming files, there is a certain degree of complexity in tagging

  svn copy https://svn.cern.ch/reps/glitedm/trunk/gfal \
    https://svn.cern.ch/reps/glitedm/tags/glite-data-gfal_R_1_11_12_3 \
    -m glite-data-gfal_R_1_11_12_3

compared to CVS

    cvs tag glite-data-gfal_R_1_11_12_3

or git

   git tag glite-data-gfal_R_1_11_12_3

FTS releases

The current, 'WLCG approved' version of FTS is FTS 2.1.

gLite has already released FTS 2.2 (i.e. FTS 2.2.0), however Atlas has
discovered some shortcomings with the checksum suppport in the
Pilot service, which were fixed in the upcoming FTS 2.2.1 and
FTS 2.2.2 releases.

FTS 2.2.2 has been certified by gLite and installed on the CERN FTS
Pilot service and will be running until the beginning of December
to reach the 'WLCG approved' status.

Currently we are working on FTS 2.2.3 to address a few more issues.

Have a look at the FTS patch status page for more details on older releases!

releases

My magic URL for upcoming data management releases points into Savannah.

We usually create a 'patch' (i.e. release) when we have a draft idea of what should
go into a release. For example FTS 2.2.3 and GFAL 1.11.13 are created with all the
bugs attached that we intend to implement by the release date.

The first noteworthy state is 'Ready for certification', when the developers have
finished their work and there are already RPMs created. At this point we usually
upload the packages into our Release Candidate repository for the convenience of
early testers.

The next noteworthy state is 'Certified', when the release has passed all regression
tests and the new features seem to be working.

After this state there is a few weeks of testing (i.e. waiting if there is any unexpected
behaviour) in the pre-production testbed (PPS) and then comes the gLite release.

Unifying LGC_Util and GFAL version numbers

A usual source of confusion: which LCG_Util version requires which GFAL library version. Almost after each release somebody installed the wrong packages somewhere. Now, the confusion is over: from the next release on, we always release those two components together, under the same version numbers (but with different tag prefix, certainly). We will create the first such a release pair this week, with version numbers 1.11.12-1 (the next GFAL version number). It means, that there will be a gap in LCG_Util case: version 1.7.8-1 will jump up to 1.11.2-1. Keep tuned.

Debugging tricks

When you want to debug the command-line tools of the projects, you find immediately that the commands are in fact shell scripts. They are libtool wrapper files actually and set up several things before calling the binaries themselves. You need to invoke gdb in the following way:

libtool gdb _command_

From this point, everything should work as usual.

Next, you may run into the following trouble when debugging:

[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error

For me, it occured on SLC5, when the code used the dynamic linking library (dlopen, etc.). You can eliminate the problem by linking libpthread directly to your executable. For example:

lcg_del_LDADD = $(COMMON_LIBS) -lpthread

Good luck!