rotki Contribution Guide

rotki is an opensource project so help is really appreciated.

Bug Reporting

Before reporting an issue, make sure to check the issue tracker for similar ones. If this is a new issue then use the proper template providing a detailed description about:

  • Problem: what happened and what you were expecting to happen instead.

  • Logs: run rotki in debug mode, replicate the issue and attach the logs (see the section Run rotki in debug mode).

  • Environment: the operating system and the rotki version.

Run rotki in debug mode

For running rotki in debug mode, you can do it either via a config file or the app UI:

  • Config file: see the section Set the backend’s arguments.

  • App UI: before log in, click the cog wheel at the bottom right corner and select “Debug” (image below). Press the save button and proceed to log in as usual.

Run rotki in debug mode via app UI

You can open the app logs location by going to “Help” menu at the top and then choosing “Logs Directory”.

The default log locations are:

  • Linux: ~/.config/rotki/logs

  • OSX: ~/Library/Application Support/rotki/logs

  • Windows: %APPDATA%\rotki\logs

Feature Requests

Use the feature request template.

Describe exactly what it is that you would like to see added to rotki and why that would provide additional value.

Please note that feature requests are just that. Requests. There is no guarantee that they will be worked on in the near future.

Contributing as a Developer

Being an opensource project, we welcome contributions in the form of source code. To do that you will have to work on an issue and open a Pull Request for it.

In order for your Pull Request to be considered it will need to pass the automated CI tests and you will also need to sign the CLA (Contributor’s license agreement).

Committing Rules

For an exhaustive guide read this guide. It’s all really good advice. Some rules that you should always follow though are:

  1. Commits should be just to the point, not too long and not too short.

  2. Commit title not exceed 50 characters.

  3. Give a description of what the commit does in a short title. If more information is needed then add a blank line and afterward elaborate with as much information as needed.

  4. Commits should do one thing, if two commits both do the same thing, that’s a good sign they should be combined.

  5. Never merge master on the branch, always rebase on master. To delete/amend/edit/combine commits follow this tutorial.

When pushing on a PR the tags [skip ci] or [ci skip] can be used a part of the commit message to skip the run of all the CI jobs (lint, test etc).

Lastly, we encourage using signed commits:

git config commit.gpgsign true
git config --global user.signingkey <KEYID>

For more information about signing commits, check out Verify commit signatures.

Linting / Formatting

Make sure to run make lint before pushing your commit. This runs isort, flake8, mypy and pylint on the code to make sure that formatting rules and common mistakes are not committed in the code.

Set your editor up to use isort as seen in the Makefile (format command) before saving each file.

You can also bulk apply formatting changes to all files by running make format

Where to make changes

  • If you want to contribute fixing a bug use the bugfixes branch.

  • To add new assets also use the bugfixes branch.

  • Any other change can be made against the develop branch.

Our releases work like this:

  • We release patches merging the bugfixes branch to master and adding a new tag.

  • Normal releases are created by merging the develop branch to master and adding a new tag.

Adding new assets to rotki

To add new assets for rotki you will have to edit the SQL file in the last update at the assets repository. SQL sentences for insertion differ depending on if we are adding an ethereum token or other types of assets. More information about each type of asset and columns is available at the readme file.

rotki uses CAIP-19 to uniquely identify assets. So for example, if we wanted to refer to the Uniswap (UNI) ERC-20 token on the Ethereum mainnet chain, the correct identifier would be eip155:1/erc20:0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984.

Once you have finished adding assets it would be necessary to update the file containing metadata about the update. To do so run the script:

python tools/populate_infojson.py

This will update the file info.json. Finally execute the tests to detect possible errors in the SQL sentences using:

pytest tests

In order to do so you will need to install the dependencies in the requirements.txt file.

Get CoinGecko asset identifier

In most cases the CoinGecko asset identifier matches the URL one, for example “weth” for WETH. However, sometimes it doesn’t, for example “sharering” for SHR (“sharetoken” in the URL). Lately coingecko added the API id of the asset to the information provided for the asset.

Obtain id for assets at coingecko

This identifiers mismatch can be detected by running the this test:

python pytestgeventwrapper.py -xs rotkehlchen/tests/unit/test_assets.py::test_coingecko_identifiers_are_reachable

The test warns each mismatch suggesting the potential identifier (e.g. Suggestion: id:sharering name:ShareToken symbol:shr). This identifier can be checked via the GET coins by id endpoint on the CryptoCompare API explorer.

The test also warns about any asset delisted from CoinGecko. In that case, add the delisted asset identifier in the coins_delisted_from_coingecko list.

Get CryptoCompare asset identifier

One important gotcha is to check for CryptoCompare asset prices. Unfortunately you need to to check the page of each asset in CryptoCompare. For example for $BASED you would need to check the page and then try to see the api call for USD price to see if it exists. If this returns something like:

{"Response":"Error","Message":"There is no data for any of the toSymbols USD .","HasWarning":true,"Type":2,"RateLimit":{},"Data":{},"Warning":"There is no data for the toSymbol/s USD ","ParamWithError":"tsyms"}

Then that means you have to check the CryptoCompare page and compare directly with the asset they have listed there. Like so and see that it works. Then you need to edit the CryptoCompare mappings in the code to add that special mapping here. If you don’t find your asset on CryptoCompare just put an empty string for the cryptocompare key. Like cryptocompare: "".

Hopefully this situation with CryptoCompare is temporary and they will remove the need for these special mappings soon.

Helpful commands

  • To get the checksummed ethereum address, you can get from the Python console using our code simply by doing:

    >>> from eth_utils.address import to_checksum_address
    >>> to_checksum_address("0x9c78ee466d6cb57a4d01fd887d2b5dfb2d46288f")
    '0x9C78EE466D6Cb57A4d01Fd887D2b5dFb2D46288f'
    

Adding new Centralized Exchanges (CEXes)

All centralized exchanges modules live in a separate python file under here.

As an example of how to add a new CEX you can check the Bitpanda PR.

Add Location

You should add a new value to the location Enum and also make sure that the value is mirrored in the DB’s schema as seen here. Add it also in the SUPPORTED_EXCHANGES list here. Finally don’t forget to add it in the latest DB upgrade as seen in the Bitpanda PR linked in the start of this section.

Create exchange module

To add a new CEX you should create a new file with the name of the exchange all lowercase in here.

It should have a class which should be the exact same name as the file but with the first letter capitalized. So if the module name is pinkunicorn.py the class name should be Pinkunicorn.

That class should inherit from the ExchangeInterface and implement all the required methods.

It should have an edit_exchange_credentials() and validate_api_key() to be able to validate and accept new credentials.

It should have a query_balances() to return the current balances of the user in the exchange.

It should have a query_online_trade_history() to query the trade history endpoint of the exchange for a given time range and save them in the database.

It should have a query_online_deposits_withdrawals() to query the deposit/withdrawals history endpoint of the exchange for a given time range and save them in the database.

Optionally it can have a query_online_income_loss_expense to parse any special data from the exchange that can create income/loss items for the user such as staking events.

Add Asset Mappings

Exchanges have assets listed by symbols. This is unfortunately inaccurate and has conflicts since there is no central crypto registry and there is way too many crypto assets using the same symbol.

We tackle this by having special mapping such as this one here. So you would add the mapping WORLD_TO_MYNEWEXCHANGE. Then you would create an asset_from_mynewexchange() function like this one for bittrex.

To find any assets listed in the exchange that are not mapped perfectly you would need to find and call the endpoint of the exchange that queries all assets. Then you need to write a test like this which queries all assets and tries to call the asset_from_bittrex() function. If any asset is not mapped properly a warning should be raised so we the developers figure out a new asset is added and we need to map it.

Add tests for the exchange

You should write tests for all the endpoints of the exchange you implemented. To see what tests and how to write them check the bitpanda PR linked in the start of this section.

You will generally need to:

  • Touch rotkehlchen/tests/api/test_exchanges.py::pytest_setup_exchange()

  • Add a new test module under rotkehlchen/tests/exchanges/

  • Add a new fixture for the exchange at rotkehlchen/tests/fixtures/exchanges/mynewexchange.py and expose it in rotkehlchen/tests/fixtures/__init__.py

Adding new ethereum modules

This guide is to explain how to add a new ethereum module into rotki and its corresponding transaction decoder and accountant.

Add new module directory

Each ethereum module lives in this directory. To add a new module you should make sure the name is unique and create a new directory underneath.

The directory should contain the following structure:

|
|--- __init__.py
|--- decoder.py
|--- constants.py
|--- accountant.py

Almost all of the above are optional.

The decoder

As an example decoder we can look at makerdao.

It needs to contain a class that inherits from the DecoderInterface and is named as ModulenameDecoder.

Counterparties

It needs to implement a method called counterparties() which returns a list of counterparties that can be associated with the transactions of this modules. Most of the times these are protocol names. Like uniswap-v1, makerdao_dsr etc.

These are defined in the constants.py file.

Mappings and rules

The addresses_to_decoders() method maps any contract addresses that are identified in the transaction with the specific decoding function that can decode it. This is optional.

The decoding_rules() defines any functions that should simply be used for all decoding so long as this module is active. This is optional.

The enricher_rules() defies any functions that would be used for as long as this module is active to analyze already existing decoded events and enrich them with extra information we can decode thanks to this module. This is optional.

Decoding explained

In very simple terms the way the decoding works is that we go through all the transactions of the user and we apply all decoders to each transaction event that touches a tracked address. First decoder that matches, creates a decoded event.

The event creation consists of creating a HistoryBaseEntry. These are the most basic form of events in rotki and are used everywhere. The fields as far as decoded transactions are concerned are explained below:

  • event_identifier is always the transaction hash. This identifies history events in the same transaction.

  • sequence_index is the order of the event in the transaction. Many times this is the log index, but decoders tend to play with this to make events appear in a specific way.

  • asset is the asset involved in the event.

  • balance is the balance of the involved asset.

  • timestamp is the unix timestamp in milliseconds.

  • location is the location. Almost always Location.BLOCKCHAIN unless we got a specific location for the protocol of the transaction.

  • location_label is the initiator of the transaction.

  • notes is the human readable description to be seen by the user for the transaction.

  • event_type is the main type of the event. (see next section)

  • event_subtype is the subtype of the event. (see next section)

  • counterparty is the counterparty/target of the transaction. For transactions that interact with protocols we tend to use the CPT_XXX constants here.

Event type/subtype and counterparty

Each combination of event type and subtype and counterparty creates a new unique event type. This is important as they are all treated differently in many parts of rotki, including the accounting. But most importantly this is what determines how they appear in the UI!

The place where the UI mappings happen is frontend/app/src/store/history/consts.ts.

The Accountant

As an example accountant module we can look at makerdao.

The accountant.py is optional but if existing should also be under the main directory. It should contain a class named ModuleNameAccountant and it should inherit the ModuleAccountantInterface.

What this class does is to map all the different decoded events to how they should be processed for accounting.

These accountants are all loaded in during PnL reporting.

Each accountant should implement the reset() method to reset its internal state between runs.

Event Settings mapping

Each accountant should implement the event_settings() method. That is a mapping between each unique decoded event type, identified by get_tx_event_type_identifier() and its TxEventSettings().

So essentially determining whether:

  • taxable: It’s taxable

  • count_entire_amount_spend: If it’s a spending event if the entire amount should be counted as a spend which means an expense. Negative PnL.

  • count_cost_basis_pnl: If true then we also count any profit/loss the asset may have had compared to when it was acquired.

  • take: The number of events to take for processing together. This is useful for swaps, to identify we need to process multiple events together.

  • method: Either an 'acquisition' or a 'spend'.

  • multitake_treatment: Optional. If take is not 1, then this defines how we treat it. It’s always a swap for now, so TxMultitakeTreatment.

  • accountant_cb: Optional. A callback to a method of the specific module’s accountant that will execute some extra module-specific pnl processing logic. The makerdao accountant linked above has some examples for this.

Multiple submodules

The modules system is hierachical and one module may contain multiple submodules. For example uniswap having both v1 and v3 each in their own subdirectories as seen here.

Add a new language or translation

Add new language

The translation files are located here. They are saved with format {language_code}.json. You can see the list here. If you want to add a new language, you need to create a new language file with that format, and then fill it.

You also need to update the frontend mapping that is defined at this enum, and these entries. The countries field will be used to show the countries flag on the app. You can see the list here.

Add or edit a translation

Rotki does translation using Vue i18n.

Rotki’s main language is English. The language file for it is here. In order to fill in the translation for another language, you should pay attention to the following things:

  1. The JSON structure from the English language file is absolute, meaning you can’t change the JSON structure (the keys), because this is how rotki reads which value to use. So for translations of other languages, please follow the same structure as the English language JSON file. For example:

// en.json
"exchange_balances": {
  "add_exchange": "Add exchange",
  "click_here": "Click here",
}

// es.json
"exchange_balances": {
  "add_exchange": "Añadir intercambio",
  "click_here": "Haga clic aquí",
}

2. You may notice that there are some words that are wrapped inside curly brackets, for example the word length in the sentence Use total from {length} asset(s) value. This is how rotki inserts a variable inside a sentence. You must keep this variable name, when translating to a different language. What you can do though is to reposition the variable inside the sentence. For example:

// en.json
"total": {
  "use_calculated_asset": "Use total from {length} asset(s) value: ",
}

// es.json
"total": {
  "use_calculated_asset": "Utilice el valor total de {length} activos: ",
}
  1. For missing keys from other language files, by default it will use the value of the master file which is English.

Code Testing

Python

In order to run the python test suite, first make sure the virtual environment is activated, the developer requirements are installed, and then do:

python pytestgeventwrapper.py -xs rotkehlchen/tests

We require this wrapper as a drop-in replacement of pytest due to quirks of gevent and monkeypatching.

For running the tests with a more specific usage and invocation, please refer to the pytest documentation.

Linting

Before each commit you should run the linting checks. They run flake8, mypy and pylint in order.

Do that by invoking make lint from the root directory of the project.

Alternative Linting and Static Analysis Tools

There is some alternative linting tools that we don’t run in the CI since they have a lot of false positives. It’s good to run them from time to time so they are listed here.

  • vulture: Source and docs here. Just get via pip install vulture. If you simply run it from the root directory you will get a list of possibly unused code that you can remove. You will have to go through a lot of false positives.

  • bandit Source and docs here. Just get via pip install bandit. If you run it you will get a lot of potential issues in the code. You will have to go through a lot of false positives.

Vue/Typescript

The Vue/Typescript part of the application under the frontend directory has two types of tests. The unit tests that are testing functions and components are using vitest and vue-test-utils and you can run them by:

npm run test:unit -w app

These are supposed to be small tests ensuring that parts of the code work good in isolation.

The second type of tests is an e2e test suite using cypress. The e2e tests require the python virtual environment because they depend on the actual python backend. These tests ensure proper e2e functionality and application integration and try to replicate scenarios of real user interaction through the application.

To run the e2e tests you need to run the following command inside the frontend directory:

npm run test:integration-ci -w app

The above command will run the e2e tests in headless mode. If you want to debug specific tests you can also run:

npm run test:integration -w

This command will open the Cypress Test Runner window where you can select specific suites to execute.

Linting

Before committing and pushing your commits ensure that you fix any lint issues. You can do this by running:

npm run lint:fix

Note

While lint warnings are not fatal and will not fail the CI pipeline it would be better if a PR reduces the number of warnings and doesn’t introduce new ones. Warnings are things that need to be fixed and they will be converted to errors in the future.

Manual Testing

In order to make sure that the final executable works as a complete package (including the UI) a bit of manual testing with the final binaries is required.

This should eventually be reduced when we manage to have a more complete E2E test suite. Everything below that can be E2E tested should be.

If time allows test the below on the binaries for all OSes. If not just on one.

Startup

New User

  • Create a new user and see that it works. Both with and without a premium key. With a premium key make sure that you can verify that pulling data from the server works.

  • Provide mismatching passwords and see it’s handled properly.

  • Provide wrong premium keys and see it’s handled properly

Sign in existing user

  • Sign in an existing user with a wrong password and see it’s handled.

  • Sign in a non-existing user and see it’s handled

  • Sing in an existing user and see it works

External Trades

  • Add an external trade and see it’s added in the table

  • Edit an external trade from the table and see it’s altered

  • Delete an external trade from the table and see it’s removed

  • Expand the details on a trade and see they are shown properly

Data Importing

  • Import some data from cointracking.info and see that works properly

Exchanges

  • Add an invalid exchange API key and see it’s handled properly

  • Add a valid exchange API key and see it works. See that dashboard balances are also updated.

  • Remove an exchange and see that it works and that the dasboard balances are updated.

External Services

  • Add an API key for all external services

  • Remove an API key for all external services

Application and Accounting Settings

  • Change all application settings one by one and see the changes are reflected.

  • Same as above but for invalid values (if possible) and see they are handled.

  • Change the profit currency and see it works

  • Change all accounting settings one by one and see the changes are reflected.

  • Same as above but for invalid values (if possible) and see they are handled.

Accounts and Balances

Fiat

  • Add a fiat balance and see it works

  • Remove a fiat balance and see it works

  • See that adding non number or negative is handled

Ethereum Accounts

  • Add an ethereum account and see it works

  • Add an invalid ethereum account and see it is handled properly

  • Remove an ethereum account and see it works

  • After adding tokens to an account that has it expand the account and see all tokens owned by it are shown.

Ethereum Tokens

  • Track an ethereum token and see it works. Works is defined as being added:
    • In the dashboard

    • In the owned tokens

    • In total blockchain balances

    • In the expanded asset details of ETH accounts that own it.

  • Remove an ethereum token and see it works. Works means being removed from all the above.

Bitcoin accounts

  • Add a bitcoin account and see it works

  • Add an invalid bitcoin account and see it is handled properly

  • Remove a bitcoin account and see it works

Tax Report

  • Check that invalid input in the date range are handled properly

  • Create a big tax report over many exchanges for a long period of time and see that it’s correct and no unexpected problems occur.

  • Create a CSV export of the report and see it works

Premium Analytics

  • Check they work for a premium account

  • Modify the range of the netvalue graph and see it works properly

  • Change the asset and modify the range of the graph of amount and value of an asset and see it works properly

  • Check the netvalue distribution by location works properly

  • Check the netvalue distribution by asset works properly and that you can modify the number of assets shown in the graph

Updating the documentation

rotki is continuously changing and sometimes documentation gets outdated. One way to contribute to rotki is by helping to keep the documentation up to date. To do so you have to edit the corresponding section in the .rst files inside the docs folder of the git repo.

To review your changes you can compile the documentation using the command

make html

inside the docs folder.

Guide Screenshots

When updating the user guide documentation you might need to update the application screenshots.

Capturing screenshots

In order to be consistent, you can use the chrome developer tools in the electron application to capture the screenshots.

First you have to toggle the device toolbar (1).

If this is the first time you are taking a screenshot, click on the Dimensions dropdown menu and select edit (2).

Adding a custom screen resolution

There you will be given the option to Add custom device. Use the following settings:

  • Resolution: 1280x894

  • DPR: 1.3

In the user agent make sure that Desktop is selected. Then proceed to save the entry.

After making sure that this entry is selected, you can press the overflow menu (3) and select the Capture Screenshot entry to capture a new screenshot.

Code profiling

Python

Flamegraph profiling

In order to use the flamegraph profiler you need to:

  1. pip install -r requirements_profiling.txt to make sure you have the latest dependencies required for profiling

  2. Install the flamegraph package in your system. Some OSes such as Archlinux have ready-made packages.

Then in order to profile a test run all you need to do is add --profiler=flamegraph-trace to the pytest arguments.

Once the test concludes this will add a data file under /tmp with the data generated by the run. Example: /tmp/20211127_1641_stack.data

Then you can run the flamegraph tool on that data to generate an svg. Example:

flamegraph.pl --title "rotki-test graph" /tmp/20211127_1641_stack.data > profile.svg

Finally open the svg with any compatible viewer and explore the flamegraph. It will look like this:

A flamegraph profiling example

rotki Database

rotki uses two different sqlite databases, one with information about assets, price and other non-sensitive information (global.db) and one with user information (rotkehlchen.db). The latter is encrypted using an extension called SQLCipher that provides transparent 256-bit AES full database encryption.

Database Location

Databases are stored in directories under the rotki data directory.

The global database is stored at global_data/global.db.

The accounts you create in rotki have their own database stored at <account_name>/rotkehlchen.db.

Exploring the database

To open the user database you can use sqlitebrowser. It supports sqlcipher and will ask for the password used to decrypt the database.

If you prefer the command line instead, you can use the sqlcipher cli tool. Note: using just sqlite3 cli will not work since the database is encrypted.

Note to Debian and Ubuntu users: we are using SQLCipher encryption v4, therefore a recent version of sqlcipher is required. Unfortunately the version available in the distribution repositories are too old and won’t let you open the database using sqlitebrowser nor sqlcipher. On top of that, the sqlitebrowser version provided is not compiled with sqlcipher support. As a workaround, you can find a PPA (use at your own risk) to install more recent versions of both packages and you can also recompile sqlitebrowser with sqlcipher support following this stackoverflow thread.

When using sqlcipher, you need to specify the password to decrypt the database entering PRAGMA key='your-secret-key'; right after opening the database.

DB Upgrades

Database upgrades are needed when changes in the schema happen. rotki checks a setting in the database with the version and just executes sequentially a check against the version to verify if the upgrade needs to happen or not.

When the database schema is changed, it is important to note that the operation is not reversible. Therefore in order to open the upgraded database with an older version you would need to have a backup. For more information, check upgrade_manager.py.

When adding a new upgrade, remember to bump ROTKEHLCHEN_DB_VERSION in settings.py. Generally we only make one upgrade per release, so if you need to make changes to the schema, simply add them to the latest unreleased migration.

rotki generates a backup before any schema upgrade. These backups are stored in the same directory as the database with name <timestamp>_rotkehlchen_db_v<version>.backup or <timestamp>_global_db_v<version>.backup.

rotki uses the same mechanism of updating the schema for both the global and the user databases.

DB Migrations

When developers need to make changes in the data but the schema does not change, a data migration is made instead. This operation can be a simple task such as deleting old backups files, inserting some rows or running a background task to update some table. In this case, the database can be opened using the previous version of rotki. For more information, check data_migrations <https://github.com/rotki/rotki/tree/develop/rotkehlchen/data_migrations>__.

Docker publishing (manual)

If a need exists to publish on hub.docker.com then the following steps need to be followed.

Note

Make sure that you are logged with an account that has access to publish to docker.

This installs the qemu binaries required to build the arm64 binary and uses buildx to build the images. Please replace the the REVISION with the git sha of the tag and the ROTKI_VERSION with the tag name.

docker pull tonistiigi/binfmt:latest
docker run --rm --privileged tonistiigi/binfmt:latest --install arm64
docker buildx create --name imgbldr --use
docker buildx inspect --bootstrap --builder imgbldr
docker buildx build --build-arg REVISION='git sha' --build-arg ROTKI_VERSION=vx.x.x --file ./Dockerfile --platform linux/amd64 --platform linux/arm64 --tag rotki/rotki:vx.x.x --tag rotki/rotki:latest --push .

Working with the frontend

While working with the frontend code and type errors in the code will be displayed inside the page. To make clicking the errors open in your editor or IDE you need to set the LAUNCH_EDITOR environment variable in your system.