Description: This is a project to provide an interactive command line interface to Bitcoin. It is not meant to do Bitcoin mining, or provide a wallet or any other monetized type of service. It is intended to allow a way to examine and tinker with the way Bitcoin and it's blockchain work. Either with your own test data or accessing the actual real time blockchain.
Update 08/08/21 I think I mentioned that using bitcoinj to read the Blockchain seemed to process it in a reasonable amount of time. However, it does slow down considerably toward the end. I wondered what the reason might be. Intuitively it would probably just be more data being handled as time goes on. Confirmation of that seemed a reasonable step to take. jconsole seemed to show memory is not a constrained resource.
So onto looking at how much I/O is being done. Sort of surprisingly there seemed to convenient, easy, way to do this. JMX didn't appear to have anything. dtrace, and dtrace derived, seemed a common recommendation but on OS/X currently appears to require disabling System Integrity Protection. Something I preferred to avoid.
Googling turned up a hit that reminded me of something I had done a long time back. I couldn't find it on my current or old machine but did get Googled hits that allowed me to come up with this from scratch...
#include <sys/types.h>Oh yeah, still have those mad C skills. Well mostly from scratch the invocation is pretty much copied to be honest.
#include <unistd.h>
#include <stdio.h>
#include <libproc.h>
#include <sys/resource.h>
main()
{
rusage_info_current rusage;
pid_t ppid = getppid();
if (proc_pid_rusage(ppid, RUSAGE_INFO_CURRENT, (void **)&rusage) == 0)
{
printf("%llu",rusage.ri_diskio_bytesread);
}
}
I didn't even bother to JNI it. Just Runtime exec'd. So discretizing by blocks I was able
to come up with this stats.csv. I did some simple ggplot's in R.
For both disk reads and elapsed time the plots start out rather smoothly increasing. There
is a tipping point for both where the scatter plot gets more scattered. The smoothness is
no longer there. Also, there seemed to be some differences in how this dispersal occurs
so I plotted the reads and elapsed together scaled...
I think it can be seen that not only aren't reads and elapsed smooth anymore, they also no longer move
as tightly together. Maybe roughly still having the same direction changes but to different
amounts? I'm not sure why this would be. Why would elapsed time no longer be pretty much
strictly a function of I/O reads? Possibly some resource constraint affecting the java code?
But jconsole had seemed to not indicate any memory issues so what constraint?
Possible reasons for the general loss of smoothness might be a block size constraint, no way to get bigger blocks. Maybe miner's strategies change and sometimes they don't go for the maximum size. Possibly this depends on price. I thought I should put price into the data. Then I thought I should generally extract more data to include.
I did some searching and there seems to be some discretizing of price data by date. I saw no one doing it by block. Is there a fundamental difference between the two? Probably not. Both are natural ways to divide up Blockchain data. By block maybe more natural to my sequential block read approach. I haven't looked at what other data these other approaches might have included but having some block summary statistics and the like might be useful in finding some sort of leading indicators for changes. Miner strategy change signals or even general price movement changes. Something machine learning could predict off of?
That would be the BlockDiscretized but I just had time to start it.
A quick addtional update to this one. Actually looking at R correlations before and after
the tipping point...
>
[1] 0.9992126
>
[1] 0.853118
The 47 split is a guesstimate. This does show extremely high correlation to begin with that
falls off some but is still very correlated. Who knows? maybe it's hardware.
That little C snippet in case anyone wants to use it without typing it or cutting and pasting from HTML is io.c
Disk image of the project folder...
btc.dmg
Update 08/01/21 Put some of the currency support detailed below in place. I also
renamed BlockCounter.java to RunningMax.java better suiting what it currently does. Run like...
java -cp .:lib/bitcoinj-core-0.15.10.jar:lib/slf4j-api-1.7.30.jar:lib/guava-30.1.1-jre.jar:lib/slf4j-nop-1.7.25.jar:lib/bcprov-ext-jdk15on-169.jar:lib/gson-2.8.7.jar:btcscript.jar -Dcurrency=JPY RunningMax
The available currencies are...
Some of the values checked against Bitcoin Explorer
are far enough to be in error. So it will need more looking at. For some reason it seemed like EUR and GBP
need to be divided by the exchange rate instead of multiplied by it.
Bitcoin historic with currencies can be directly tested against like...
java -cp .:lib/bitcoinj-core-0.15.10.jar:lib/slf4j-api-1.7.30.jar:lib/guava-30.1.1-jre.jar:lib/slf4j-nop-1.7.25.jar:lib/bcprov-ext-jdk15on-169.jar:lib/gson-2.8.7.jar:btcscript.jar -Dcurrency=CNY us.hall.bitcoinj.BTCHistoricPrices
There is nothing in place yet to allow overriding the Federal Reserve URL that obtains the exchange rates. I'm still not sure I'll provide that yet.
Disk image of the project folder...
btc.dmg
The external drive worked without an error. Faster even, but still not real fast. It took days to download and sync the entire blockchain. But I now have it and run as a first class citizen full node. I also can now do bulk I/O directly against the blockchain and not have to strictly rely on network query API's.
I found Exploring the Bitcoin blockchain using Java using the bitcoinj API. This project already included bitcoinj so that works out fine. The fact that it is based on a utils package class suggests that the was not something in the mainstream of planned usage for the API. But it seems to work pretty decent and isn't terribly slow considering the data size.
So what is the biggest transaction that has occurred on the blockchain? This seemed a reasonably easy way to begin analysis. I think this question has to be answered in the context of the value of Bitcoin at the time. I needed historical Bitcoin price data for this. I think given the global nature of Bitcoin this should be considered in whatever currency you are most used to.
For the historical price data I am using CoinDesk Bitcoin Price Index API. I had already been using this in my HalfPipe application to show realtime Bitcoin prices in a couple different currencies. It doesn't keep historical data before 07/19/10 but then you don't look for Moby Dick in Boston Harbor. You have to get a little farther out.
For currency support I thought you might want more than the three or four that HalfPipe
provides. I wondered what countries were big into Bitcoin. This turned out to be a bit of
a surprise to me -
Which Countries Are Using Cryptocurrency the Most?
the world's most powerful countries with the biggest economies fall on the bottom half of the list.Many of these weren't currencies I planned on including. I was also a little surprised that Nigeria is the largest African economy. I tend to think of them as the origin of bad internet scams.
The world's top 10 crypto countries, according to the Statista data, are:
Nigeria: 32% Vietnam: 21% Philippines: 20% Turkey: 16% Peru: 16% Switzerland: 11% India: 9% China: 7% U.S.: 6% Germany: 5% Japan: 4%
One Country Stands Out Above All the Rest
Africa's largest economy is the king of cryptocurrency. In Nigeria, 32% of respondents - nearly 1 in 3 - report having used or owned one type of crypto or another in 2020. By comparison, just 6% of Americans reported the same.
It seemed that including these currencies just can't be done. What I planned on using for
currency exchange data is the
Federal Reserve - Foreign Exchange Rates - H.10. It doesn't include Nigeria, or
Vietnam. Looking elsewhere I found current rates but not historical, anywhere. I did add
a couple currencies after seeing this list...
So how do you get prices in these currencies? You don't yet. Maybe I'll have that in by
next week so it can be used for Bitcoin prices. The class is us.hall.bitcoinj.Currencies.
It has a main, you can run it command line to output the data. It doesn't even update the
last date in the range to today yet, but it is fairly current.
I am thinking about having a system property to allow selecting the currency. Possibly even a property the user can use to indicate their own Fed URL. There are other currencies supported that I don't include.
For this week though what I have done is in US dollars. I just walked the blockchain on disk and output every new maximum in value transaction. That file is evolution_of_btc_whales.txt It starts at $4.04 believe it or not.
The latest generation of whales seem to have become visible around year end of last year...
Block height 662608: tx id 44e7255475ac7cea2da8005e8b391c783f78c55983c9cdb6acf5482d2f059180: 2020-12-24T21:43:28Z $2,266,442,236.13
Block height 662981: tx id 3764b076ad118d89338e606d54b41de836b05b01ed5ed526c40f31eb79005f79: 2020-12-27T09:56:10Z $2,388,881,676.01
Block height 663709: tx id 1b98a0731e8b7bbac300a4703f516f8f7224d97aeb56939e115da4a5078ad29f: 2021-01-01T10:06:12Z $2,425,144,464.33
Block height 663827: tx id 606a40065c490cdb8bf01fec04f383e273f533be4aab405bc2662292fad8eeaa: 2020-12-31T13:28:36Z $2,461,600,774.97
This was the first time $2B was cracked. Being around year end makes you think maybe it was
somehow tax year related. Locking in profits or losses. I think I have heard of the same
idea related to stocks. It could be the same Bitcoin with the small differences due to day
to day variations in prices. I haven't tried to verify that.
The current largest transaction wasn't until February of this year.
Block height 671029: tx id 2fa413fed0aac4c8c85ad8aa96636bda5c30a8283dcad988f045fb395b4f01e9: 2021-02-19T22:46:51Z $6,040,312,436.22
I believe there is some analysis of this transaction
Power Transaction: Bitcoin Whales Move Over $6B Worth Of Crypto In A Day. Although
somewhat strangely it indicates the date for that as April 22. It perhaps involved more than
one transaction whereas mine was a single $6B transaction also shown on
blockchain Explorer. The amount differs somewhat from mine based on whatever they use
for price history versus what I use and the time is also somewhat different based on whatever
they use for dates. For what I display here I use a bitcoinj utility. It is the same
transaction as you can verify by cutting and pasting the "tx id" or hash. If I come across
an application where timing needs to be more precise I may need to better understand what
is correct.
When I first saw a $1B transaction I said "Holy moly Rocky". After I saw the $6B transaction I thought the code had a bug. I wondered how this would compare to your banks in asset size. Biggest US Banks by Asset Size (2021). Interestingly there are also indicated to be whales in this industry. (Remember "too big to fail"? Good times).
For years, JPMorgan Chase, Bank of America, Wells Fargo, and Citi have dominated the industry, and this year is no different.I have seen that more institutional investors are moving into Bitcoin. Usually though these large transactions have been attributed to Bitcoin exchanges. Clearly these exchanges are managing assets valued considerably more than most banks do. There are getting to be real whales moving a lot of value across the Bitcoin blockchain wire.
...
The average asset size is $4 billion, largely driven up by the biggest four. (For comparison, the average credit union has an average of $286 million in assets.)
...
The median asset size, by contrast, is $241 million.
Invocation of the maximum transaction logger. Path to Bitcoin core blocks directory is
hardcoded, if you are running a full node.
java -cp .:lib/bitcoinj-core-0.15.10.jar:lib/slf4j-api-1.7.30.jar:lib/guava-30.1.1-jre.jar:lib/slf4j-nop-1.7.2jar:lib/bcprov-ext-jdk15on-169.jar:lib/gson-2.8.7.jar:btcscript.jar BlockCounter
Disk image of the project folder...
btc.dmg
UPDATE 07/04/21 Review for continuing
I think I never really knew exactly what I wanted to do with this project. I also remember
running into problems in handling scripts related to OP_CHECKSIG. Getting the crypto check
right as best as I can still remember. Left the code very messy.
I have been thinking for a while about doing some additional coding related to bitcoin. This seemed as good a place to start again. Actually, for one thing, the code appears to be a usable early effort at a command line implementation of the blockchain.info API's [8]. So for now I have been doing a little to improve it as such. Get it to work and look more like the browser implementation.
It is my intention to do some other bitcoin related things that this API won't be able to support. More serious crawling of the blockchain. For this I am assuming I will need the full client and blockchain on disk. Currently 350GB according to Bitcoin Core Requirements and Warnings. I've ordered some external storage.
Current download OS/X dmg only
Ant builds, currently running with...
java -cp btcscript.jar:lib/bcprov-ext-jdk15on-169.jar:lib/api-2.0.2.jar:lib/bitcoinj-core-0.15.10.jar:lib/gson-2.8.7.jar us.hall.btc.Script
There is supposed to be some information with 'help' for the blockchain query api but that
appears broke at the moment. You would have to look at the source for Script.java to get
a better idea of what is available.
I noticed this in the text entry for the browser api...
Search your transaction, an address or a block
I thought it was a nice feature where the code would figure out what you were looking for based on
what you enter. I implemented a feature where you can enter the block height and it will
return the information for the main chain block at that height. Testing something like...
SCRIPT>getlatest
get the current block height from...
Block: Height - 689704
and enter...
SCRIPT>689704
The block information should follow.
Better documentation and code hopefully, also, to follow. Determining if an address or transaction is entered included.
UPDATE 11/01/15 Still looking at this. I had put some code in place for OP_CHECKSIG but it didn't verify. What to do? Step by step checking my results against bitcoinj seemed like a reasonable way to proceed. So I started looking at that.
It is a considerably different API with what seems like a little more learning curve. There might be other benefits beside cross-testing to combining the api's. I found out bitcoin supports separate test networks, blockchain.info I don't think supports these but bitcoinj does. Maybe you could use something like the simpler blockchain.info api to use the test networks?
Anyhow I've got so far as figuring out how to get the latest block from bitcoinj using their wallet kit, this based on a forum post I came across on bitcoin.stackexchange. My slightly modified version DumpLastBlock.java. Given this I should be able to get the same transaction I'm testing blockchain.info with and go from there.
I am not providing a full update until this is a little farther along. It does involve some new jars to go with the bitcoinj code. For example, apparently bitcoinj decided to cut over to spongycastle instead of bouncycastle for some or all of their crypto?
There should be more to come eventually.
UPDATE 10/19/15 A little progress. For a typical single input and output script
I can get through the OP_EQUALVERIFY. This means successfully hashing the input public
key and using it to compare against the script with the correct embedded
In response to a github issue the blockchain.info java api added a class from bitcoinj to handle Base58 decoding. I'm trusting them that the licensing issues aren't a concern. The code works fine. I also took a look at the bitcoinj Script.java code to figure out how to correctly handle my own OP_HASH!60 implementation. OP_CHECKSIG still doesn't work and should be next. Then more complex scripts. Then...
UPDATE 10/11/15 Got into transactions. Starting to look at running scripts with
associated crypto for the transactions. A lot of debug output related to that right now -
because it's totally buggy.
New commands...
'getlatest full' is 'getlatest detail' plus transactions. A lot of output.
'getlatest tx' returns the first transaction from the latest block.
'tx next' return the next transaction from the latest block.
'tx next verify' attempts to crypto verify by running the script of the next transaction
in the latest transaction.
I started with 'getlatest tx verify' to verify the first transaction but that is the block
'coinbase' transaction that creates new bitcoins awarded to the block posting miner.
Nothing else to the point it's worth mentioning yet.
UPDATE 10/04/15 The 'getlatest' command is currently all I support for the blockchain.info data api's. This update shows a little more information for the latest bitcoin block when using that. Making it 'getlatest detail' instead gives you a little more information yet. There is nothing for transactions yet. I still consider that the important step before we start handling scripts or anything else in anything more than a 'toy' sense. Making sure our crypto works with real transactions, etc.
The main motivation for starting this effort is that I am currently taking the Coursera/Princeton course on Bitcoin. It is an early first attempt at the course and some things seem not to have been quite in complete polished form at the course outset. One thing was that no programming assignments were included. I thought as the course went along I would see if any areas did seem to provide opportunities to write a little code.
A quick first attempt was to take the latest data from Block Explorer[2] parse out the times blocks were posted to the blockchain. Then, read that into R and plot a histogram to see if the times appeared to indicate the exponential function indicated in lecture. After omitting a few outliers I didn't try to track down, the times appeared to follow this function very nicely to me.
One interesting possibility appeared to be to try to do something with Bitcoin Script[3]. I had never heard of this before taking the course. It seems Bitcoin transactions all come with small embedded scripts that indicate both their purpose and how they are to be handled. These are based on a small Forth-like computer language. I still remember a little bit of coding MacForth from the early days of the Mac 128K's, so found this interesting in itself. Last week I wrote a really simple, incomplete, interpreter for Bitcoin script. I am using code from BouncyCastle[4] for the crypto parts of the Bitcoin processing. Otherwise, my initial attempt was largely based on this Follow The Coin [5] post.
The most common scripts simply transfer Bitcoin from one party to another party. The final step from that, implementing the script OP_CHECKSIG[6] opcode is still incomplete. This requires getting into the ECDSA signing part of the crypto and having valid public keys and signatures. [NOTE: my current intention is to base this part on this code[7] posted to a Bitcoin forum.
Another site mentioned a number of times in the Coursera course is bitcoin.info[8]. This appeared to provide a very nice and complete api related to Bitcoin and the blockchain. The api's are described here[9]. I am currently looking to use parts of both the data and simple query api's. I include the complete java api from github[10].