If your main limitation is the memory you can make use of old used server machines (approx. 6-10 years old). These machines usually utilize cheap DDR3 server memory chips and support quite impressive capacities (in the range of hundreds of GBs). On the other hand CPU speed is not that impressive even though they are usually dual processor and sport from 8 to 24 cores total.
Take a look at HPE DL380e gen8 or HPE DL380p gen8, these are fairly common models which I am familiar with. You can find several reddit threads where people discuss setting up home labs (mostly for storage rather than computation) which can be helpful for you.
For about 1000 USD you can probably buy a really powerful machine, especially since you do not need a lot of hard drives which is the main cost of a home lab. Just be sure to consult the documentation to buy the right memory chips (e.g. here).
Also, if you want to go further, you can create a cluster of such machines. Equip them with cheap used 10Gb/25Gb/50Gb/100Gb ethernet cards, preferably with ROCE, and you can run a nice cluster. If you want a 3 machine cluster, then if you buy two port network cards, you can connect machines directly without a network switch. Otherwise, you would need to buy a used (or brand new) 10Gb/25Gb/50Gb/100Gb switch which can be more costly.
Last note: beware, these machines can get really loud. They are meant to be placed in a server room, not your living room. The noise will kill you ;-)