<!--
% \VignetteIndexEntry{The AnnotationHubData Package}
% \VignetteDepends{AnnotationHubData}
% \VignetteEngine{knitr::knitr}
-->

```{r style, echo = FALSE, results = 'asis'}
BiocStyle::markdown()
```

# The AnnotationHubData Package

**Package**: `r Biocpkg("AnnotationHubData")`<br />
**Authors**: `r packageDescription("AnnotationHubData")[["Author"]] `<br />
**Modified**: 11 October, 2014<br />
**Compiled**: `r date()`

## Overview

The AnnotationHubData package provides tools to acquire, annotate,
convert and store data for use in Bioconductor's AnnotationHub. BED
files from the Encode project, gtf files from Ensembl, or annotation
tracks from UCSC, are examples of data that can be downloaded,
described with metadata, transformed to standard Bioconductor data
types, and stored so that they may be conveniently served up on demand
to users via the AnnotationHub client.

This document will deal with the nuts and bolts of how the back end of
the AnnotationHub works and operates.

## How are recipes created and then run?

For the 1st part of the description of how to put new data into the
AnnotationHub, please see the vignette titled "How to write recipes
for new resources for the AnnotationHub" in the AnnotationHub package.
This will explain how to create a recipe and an AnnotationHubMetadata
generating function.

After those initial two steps though, you will have to actually run
the recipe.  This will require that you follow the instructions
described here.

### Calling the makeAnnotationHubResource helper

Once you have a recipe and an AnnotationHubMetadata generating
functions you will need to call the makeAnnotationHubResource function
to do some automatic setup.  This function only has two required
arguments.  The 1st is basically the name of a class that describes
the kind of resource you are writing code to import.  It just needs to
be a unique class name and will be used internally to create a class
for dispatch.  The 2nd argument is the name of your metadata
processing function from the 1st step in the other vignette.  If this
function name is NAMESPACED in another package (IOW if it's a
contributed recipe) you might have to import it for the next step to
work.

```{r, makeAnnotationHubResource, eval=FALSE}
require(AnnotationHubData)
makeAnnotationHubResource("Inparanoid8ImportPreparer",
makeinparanoid8ToAHMs)
```

Once you have added this call to AnnnotationHubData, the only step
left is to export the class name into the NAMESPACE (remember that
this is that string you are providing as your 1st argument). Then
reinstall AnnotationHubData and go to the next step.

### Testing your new recipe

Next install the modified AnnotationHubData package and see if the
recipe works correctly. You just need to call the function that will
try to update the resources for your recipe.  That function is called
updateResources(). The updateResources() function takes several
arguments.  These are:

* ahroot: represents the place where you would like any generated
  files go.
* BiocVersion: character vector for versions of Bioc your objects will
  be compatible with
* preparerClasses: vector of which classes you want to process recipes
  for.
* insert: try to insert metadata into the DB? As a recipe author, you
  don't get to do this.  So it should be FALSE when you call this or
  you will get an error.
* metadataOnly: should the recipes be run to create the processed
  files?
* filtering: this is normally TRUE to indicate that we don't want to
  regenerate AHMs for records that are already in the local cached
  database.  However, you sometimes may want to set this to FALSE if
  you need to do some testing.

```{r, updateResources, eval=FALSE}
library(AnnotationHubData)

mdinp = updateResources(ahroot=file.path(getwd(),"Inparanoid"),
                        BiocVersion=biocVersion(),
                        preparerClasses = "Inparanoid8ImportPreparer",
                        insert = FALSE,
                        metadataOnly=TRUE)
```

When you call updateResources() like this, a set of
AnnotationHubMetadata objects should be returned.  You should inspect
those to make sure that they contain all the metadata that you
intended to use.  And if you run it again with metadataOnly set to
FALSE, then it should generate the final objects for you in the ahroot
directory that you specified.  If both of these things happen to your
satisfaction then your recipe is ready to be run.

## How is the back end metadata stored?

When a recipe is run a lot of metadata is also required and collected
about the data that is processed by that recipe.  This data is stored
as a record in a MySQL database back end.  When the user runs their
client, this database is downloaded (if needed) as a sqlite dump of
that MySQL backend.  Performance benefits from using MySQL come into
play whenever we need to upload new records into the back end, but it
was also decided that the portability of a sqlite DB is better for
local cached access, so both kinds of databases are used in this
implementation.  But the 'parent' database is the MySQL db (the sqlite
ones are just cached copies of that).

Normally when we work with this database we will 1st do testing on a
local instance on "MachineName" and then update the production machine
later.

To actually push data to this local database you need to set an option
in your .Rprofile for where the json should be posted to.  That should
look like this:

```{r}
options("AH_SERVER_POST_URL"="http://<MachineName>:9393/resource")
```

To log in to the database on "MachineName"" use this (you will need the
ahuser password):

```{}
mysql -p -u ahuser annotationhub
```

\begin{verbatim}
  mysql -p -u ahuser annotationhub
\end{verbatim}

And for production you will need to connect here:

```{}
ssh ubuntu@annotationhub.bioconductor.org
```

And then connect to the database:

```{}
mysql -p -u ahuser annotationhub
```

Whenever you have updated the MySQL back end, a cron job should do the
job of creating the locally cached sqlite database from the MySQL
source db.  This is done by a ruby script called convert_db.rb.  This
file in part of code that is checked in at the following URL in
github.

```{}
https://github.com/Bioconductor/AnnotationHubServer3.0
```

### How do we update the production database?

The process of updating the production server should follow this
process:

#### run the recipe on gamay to generate and files and to insert into the local copy.

You can run the recipe by calling updateResources with the insert and
metadataOnly arguments set up like this:

```{r, eval=FALSE}
mdinp = updateResources(ahroot,
                        BiocVersion,
                        preparerClasses = "Inparanoid8ImportPreparer",
                        insert = TRUE,
                        metadataOnly=FALSE)
```

#### Test the AnnotationHub locally by pointing it to gamay like this:

See section below titled "Final test that it works on gamay"

#### Make a dump of the DB from production and then copy it over to/set it up on gamay.

For this step you need to log in, and back up the database on
production.  So that should look like this (use the ahuser password
for this).  Instead of 2014_11_05, use the current date.

```{}
ssh ubuntu@annotationhub.bioconductor.org
cd ~/AnnotationHubDumps/prodDumps/
mysqldump -p -u ahuser annotationhub | gzip > dbdump_2014_11_05_fromProd.sql.gz
```

Then copy it down (from gamay):

```{}
cd /mnt/cpb_anno/mcarlson/AnnotationHubDumps/prodDumps/
scp ubuntu@annotationhub.bioconductor.org:/home/ubuntu/AnnotationHubDumps/prodDumps/dbdump_2014_11_06_fromProd.sql.gz .
```

And then unpack it on gamay (use the MySQL root password for this):

```{}
mysql -p -u root
```

Then in SQL drop and create the DB

```{}
drop database annotationhub;
create database annotationhub;
\q
```

At this point there is no data in the database, so AnnotationHub is
not working! So be sure and do the next step right away.

Then unpack the dump into the DB

```{}
zcat dbdump_2014_11_05_fromProd.sql.gz | mysql -p -u root annotationhub
```

#### Rerun the recipe on gamay to insert the rows into a clean copy of the data.

```{r, eval=FALSE}
mdinp = updateResources(ahroot,
                        BiocVersion,
                        preparerClasses = "Inparanoid8ImportPreparer",
                        insert = TRUE,
                        metadataOnly=FALSE)
```

### Final test that it works on gamay

The 1st part of this is that the server on gamay has to have produced
the .sqlite version of the mySQL database we just modified.  To make
that happen you can start by 1st impersonating dan and go to his
directory with the ruby scripts:

```{}
sudo su - dtenenba
cd AnnotationHubServer3.0/
```

From there you can look at his cron job to see how it runs every 10
mins...

```{}
crontab -l
```

But we don't want to wait ten minutes! So run the following ruby
script to make the DB right now.

```{}
ruby convert_db.rb
exit
```

Sometimes if you are rolling back there may be an issue where the ruby
conversion script doesn't run because it thinks the new DB has an
older timestamp.  If that happens remove the timestamp file and run it
again:

```{}
rm dbtimestamp.cache
ruby convert_db.rb
exit
```

Once this is done, you then have to actually test the client and make
sure that your data is in there etc. This means that you need to
launch AnnotationHub so that it points to gamay for metadata.  You can
do that like this:

```{r, test_client, eval=FALSE}
options(ANNOTATION_HUB_URL="http://gamay:9393")
library(AnnotationHub)
ah = AnnotationHub()
length(ah) ## should be larger than you started with
tail(ah) ## should have some new records
```

To manually test metadata you could just do this:

```{r, test_client2, eval=FALSE}
foo = mcols(ah)
```

And then Look for your data in 'foo'...  It's also a good idea to try
and download that data with it's "AHID"

#### Dump the database and copy it to production

```{}
mysqldump -p -u ahuser annotationhub | gzip > dbdump_2014_11_05_fromGamay.sql.gz
```

Then copy it up from gamay back to production:

```{}
scp /mnt/cpb_anno/mcarlson/AnnotationHubDumps/gamayDumps/dbdump_2014_11_05_fromGamay.sql.gz  ubuntu@annotationhub.bioconductor.org:/home/ubuntu/AnnotationHubDumps/gamayDumps/
```

And then uppack it (use the MySQL password for this):

```{}
mysql -p -u root
```

Then in SQL drop and create the DB

```{}
drop database annotationhub;
create database annotationhub;
\q
```

Then unpack the dump into the DB

```{}
zcat dbdump_2014_11_05_fromGamay.sql.gz | mysql -p -u root annotationhub
```

## Where is the actual file (cloud) data stored?

Unlike the metadata, the data that it stored after a recipe is run is
stored in something that looks like a file system. When testing on
<MachinName> this data is stored in /var/FastRWeb/, but our production server
stores and retrieves data from S3.  So to get the data moved up to
production you need to have access to these S3 resources.

To copy things up to the S3 buckets you need to use the aws command
line tools. You can see help like this

```{}
aws s3 help
```

And you can get "copying" help like this

```{}
aws s3 cp help
```

And you can copy something to be read only by using the --acl argument
like this:

```{}
aws s3 cp --acl public-read hello.txt s3://annotationhub/foo/bar/hello.txt
```

To sign in at the AWS console go here: (use your username and password
the account is bioconductor)

```{}
https://bioconductor.signin.aws.amazon.com/console
```

And then click 'S3', 'annotationhub' etc.

And to recursively copy back down from the S3 bucket do like this:

```{}
aws s3 cp --dryrun --recursive s3://annotationhub/ensembl/release-75/fasta/ .
```
And then to copy back up you could do it like this:

```{}
aws s3 cp --dryrun --recursive --acl public-read ./fasta/ s3://annotationhub/ensembl/release-75/fasta/
```

But that will copy everything (not very efficient since some things
may be the same and already there) So to actually copy back we should
really use 'aws s3 sync' instead of 'aws s3 cp'.

```{}
aws s3 sync help
```

## The AnnotationHub Server

The server is implemented in Ruby. Its code is
[here](https://github.com/Bioconductor/AnnotationHubServer3.0)
with a README that explains how to install it.

(more to come...)
