Common Questions
Development
Q: How do I start my app? What do I do now?
The easiest way to get an intuition for app development is to follow the tutorial
See if there is an existing app in the App Catalog that is similar to yours, or one that has a UI that you would like to emulate. Every app you see in the narrative will have a link to its code repository.
Typically developers follow this path:
Work on the
MyModule.specfile. This will autogenerate methods in yourMyModuleImpl.pyfile, which is where the core of your method functionality will reside.Work on the functionality of the app in the autogenerated
MyModuleImpl.pyfile.Continue developing the app by working on unit tests
Define the UI by modifying
spec.jsonanddisplay.yamlfilesPublish and test in the narrative
For these example apps, check out the .spec files, <module>Impl.py files, and spec.json files.
Q: How can I tell if someone’s already wrapped a tool I’m interested in?
Go to the development App Catalog and start by using Search to see if there’s a released method. If you don’t find a method that way, it may also help to by selecting “Organize by”… “Category” and selecting what seems to be a likely category for the tool. There are also methods that have not yet been officially released, so you should also check the “Beta” and “Dev” categories to see if there’s something in the pipeline by selecting them from the “Version” dropdown.
It may be the case that someone is wrapping a tool but is doing so in a way that doesn’t serve your needs exactly. Feel free to rewrap the tool using your approach or ask the previous tool wrapper to tweak their implementation to expose the parameters or other functionality you are looking for.
Q: What languages can I use to write my app?
We recommend Python, but you can also use Java or Perl. The language versions are as follows:
Python: 3.6
Java: 1.7.0_80
Perl: v5.18.2
KBase supports Perl and Java but the examples in the tutorial are both in Python. Here are some example modules for the other two languages:
Perl
Java
Q: How do I work with test and reference data?
Put simple data into the /data directory of your app’s repository. If the data is too large to host on Github, check out the following guide for how to add reference data.
Test files can go within the test/ directory in your app, such as test/data/.
Note
Upload local genome files for of your tests. If you use an existing workspace reference in your tests that works in AppDev, it won’t work in CI or production.
Q: How do I organize my app’s code?
There are a couple common patterns:
Create multiple functions and do everything inside the implementation file. An example of this pattern can be found in the Quast app.
Create a utils directory, create a runner or utility class, pass in the configuration file and parameter files to it, and do everything in there. An example of this pattern can be found in the FastANI app.
For more complex apps, the second option is preferred, as you can split up functionality into different modules and packages.
Q: Do you need to use GitHub? What about BitBucket? SourceForge?
You can use any public open-source revision control system. We use GitHub. The path to your repo is what you provide to the SDK Registration method to register your SDK Module.
Q: Do I need to copy the tool I’m using into my github repo?
You do not if there is a public way to retrieve the code such as by using a git clone, curl, or other way of pulling the data down into the Docker image. This is accomplished by modifying the Dockerfile to configure the Docker image build.
Q: How do I see my favorite Apps?
After logging into KBase , go to the App Catalog, and then click on the stars for your favorite Apps. You must be logged in for it to associate it with your account.
Validation
Q: How does validation work?
Currently validation is done in the UI based on values provided in the ui/narrative/methods/spec.json. When invalid input is entered in the UI for the app, an error will display to the user, and the user will not be able to submit the form for the app.
Validation is not provided for the app to be called programmatically (such as with unit tests) so you will have to validate your input again. It may be possible to generate validation programmatically using the spec.json file, but this is not currently an out of the box feature.
Q: How do I learn more about the spec.json file?
For a more exhaustive overview of the spec.json and display.yaml files, take a look at the
UI specification guide . You can also experiment with UI generation
with the App Spec Editor Narrative .
Q: What are all the auto-generated source files in my app?
These files are created by the KBase type compiler, which takes your KIDL type file (MyModule.spec) and generates different modules that are inserted into your codebase. These files are used by docker and service handlers to run your app and its dependencies.
In python apps, the authclient.py and baseclient.py files get placed into the package directory for your own app, as well as all other the package directories. These files are generally the same across all the packages in your app. authclient.py handles authenticating the user so they can access the workspace, while baseclient.py has functionality for running SDK apps and calling their methods.
Other auto-generated files in python apps include the *Client.py and *Server.py files in each package. Each app has an AppNameClient.py module in its package. These modules contain classes that can receive parameters for the app and pass that data to the app’s server
The MyModuleServer.py file gets auto-generated and saved only for your own app’s package directory. It handles requests from the python clients.
Q: What makes the tests slow to run?
Every time we run kb-sdk test it rebuilds the docker container and re-downloads data objects from your workspace.
Ways you can speed up your tests:
Make sure all your custom docker setup, such as compiling binaries, is at the top of your Dockerfile so it always gets cached
Reduce the number of files you download and upload
Reuse existing example files on the workspace so you don’t have to upload files
Separate out your modules into functions that only take local data and files, and test those separately
Q: When you install and run SDK Apps from within your own app, how are these running?
SDK app dependencies that you use in your own app, such as DataFileUtil, run in their own docker containers using their own separate servers.
When you use something like AssemblyUtil in your app, a job manager will schedule and run that app as a separate job. Despite the fact that it is run as an external job, it is not run in parallel. Your own app will hang until the other app finishes its work.
Docker
Q: Should I always specify a specific version of a library or executable?
Generally it is recommended to lock any requirements in your app to specific versions. The disadvantages of specifying versions is that you won’t automatically use newer versions of your dependencies, but the advantage is that your app will be more reliable and guards against breaking changes in its dependencies.
Q: Where do I install binaries in docker?
You can install binaries to a directory like /kb/deployment/bin and then add them to your path:
# Install Diamond Binary v0.9.17
WORKDIR /kb/deployment/bin
RUN wget https://github.com/bbuchfink/diamond/releases/download/v0.9.17/diamond-linux64.tar.gz \
&& tar -xvf diamond-linux64.tar.gz diamond \
&& rm diamond-linux64.tar.gz
ENV PATH="/kb/deployment/bin:${PATH}"
Also see the Editing Docker
Q: How do I get inside the docker image? How do I get root access?
You can open a shell inside the docker container with the test_local/run_bash.sh script.
To gain root access, remove the --user parameter or change it to --user 0
Q: What are the .sh scripts in the test_local/ directory?
You don’t really need anything besides the run_bash.sh script. If you want to run tests, use kb-sdk test.
Q: How do I have multiple docker images for different versions of my app or dynamic service?
You will need to docker build them and tag them with different names if this is not automatically done by your app.
Requirements and limitations
Q: What are system requirements for development?
You need to be able to run Docker. If you’re on a Mac, that means you must be running Mac OS X 10.8 or later. Other operating systems, such as the various flavors of Linux, are fine too. Really anywhere you can run Docker, Java, and your preferred development language (among Python, Perl, or Java). You will need about 1-2 GB free to install the dependencies and the KBase SDK
Q: Can I develop on Windows?
Sort of. Your best option right now is to install VirtualBox with Ubuntu Linux and work in the Linux VM. Many developers use this approach in KBase, and we know it works well.
Q: Can I do all the development on a remote Linux box?
Yes. All steps that require a graphical user interface are accomplished by using a web browser.
Q: What are system requirements for execution environment?
Runs completely on a standard KBase worker node (at least 2 cores and 22GB memory)
Operates only on supported KBase data types
Requires either no or fairly limited amounts of reference data
Uses existing data visualization widgets
Does not require new uploaders/downloaders
Wrapper written in Python, Java, or Perl
Q: What size data will be too big for the system?
Currently we support up to about 10 GB of accessory data for a tool (meaning reference DBs, etc). Please contact us if you need to use something larger.
As for processing, once it’s uploaded to the system (which can take a while for larger data sets), it depends on how you are using it. Currently SDK methods are limited in their memory footprint to the 22 GB of the worker nodes, so your code plus any data you load into memory must fit within that. As in any situation, we recommend the use of graceful exception handling and efficient implementations in your coding style.
Q: I need high-performance computing for my application. Is that available?
Not yet. We’re working on it!