File Database from scratch in Node Js part 2
introduction
Welcome to part 2 of trying to build from scratch and becoming a better programmer, if you just stumbled upon this post and have no idea what's going you can find part 1 here, else welcome back and thank you for your time again.
Part 1 was just a setup, nothing interesting happened really and since then I had some time to think about stuff, hence some refactor and lot of code in this part.
Database.js
Soon to be previous db function:
function db(options) {
this.meta = {
length: 0,
types: {},
options
}
this.store = {}
}
The first problem I noticed here is this.store is referenced a lot in operations.js by different functions, initially it may not seem like a big deal, but if you think for a minute as object are values by reference, meaning allowing access to a single object by multiple functions can cause a huge problem, like receiving an outdated object, trying to access a deleted value etc,
The functions(select, insert, delete_, update) themselves need to do heavy lifting making sure they are receiving the correct state, checking for correct values and so on, this leads to code duplication and spaghetti code.
I came up with a solution inspired by state managers, having a single store which exposes it's own API, and no function outside can access it without the API.
The API is responsible for updating the state, returning the state and maintaining the state, any function outside can request the store to do something and wait, code speaks louder than words, here is a refactored db function
import Store from "./Store.js"
function db(options) {
this.store = new Store("Test db", options) // single endpoint to the database
}
I guess the lesson here is once everything starts getting out of hand and spiraling, going back to the vortex abstraction and creating a single endpoint to consolidate everything can be a solution. This will be apparent once we work on the select function.
one last thing we need is to remove select from operators to it's own file, select has a lot of code
updated Database.js
import {insert, update, delete_} from './operators.js' // remove select
import Store from "./Store.js"
import select from "./select.js" // new select
function db(options) {
// minor problem: store can be accessed from db object in index.js
// not a problem thou cause #data is private
this.store = new Store("Test db", options)
}
db.prototype.insert = insert
db.prototype.update = update
db.prototype.select = select
db.prototype.delete_ = delete_
export default db
Store.js (new file)
I chose to use a class for store, you can definitely use a function, my reason for a class is, it is intuitive and visually simple for me to traverse, and easy to declare private variables
If you are unfamiliar with OOJS(Object Oriented JS) I do have two short articles here, and for this article you need to be familiar with the this keyword
export default class Store{
// private variables start with a "#"
#data = {}
#meta = {
length: 0,
}
// runs immediatley on class Instantiation
constructor(name, options){
this.#meta.name = name;
this.#meta.options = options
}
// API
// getters and setters(covered in OOJS)
//simply returns data
get getData(){
return this.#data
}
// sets data
// data is type Object
set setData(data){
data._id = this.#meta.length
if(this.#meta.options && this.#meta.options.timeStamp && this.#meta.options.timeStamp){
data.timeStamp = Date.now()
}
this.#data[this.#meta.length] = data
this.#meta.length++
}
}
Explaining setData
data._id = this.#meta.length // _id is reserved so the documents(rows) can actually know their id's
adding timeStamp
if(this.#meta.options && this.#meta.options.timeStamp && this.#meta.options.timeStamp){
data.timeStamp = Date.now()
}
// this lines reads
// if meta has the options object
// and the options object has timeStamp
// and timeStamp(which is a boolean) is true
// add datetime to the data before commiting to the db
// this check is necessary to avoid cannot convert null Object thing error
// putting the document or row
this.#data[this.#meta.length] = data
// incrementing the id(pointer) for the next row
this.#meta.length++
Now we can safely say we have a single endpoint to the database(#data) outside access should consult the API, and not worry about how it gets or sets the data
However using setData and getData sounds weird, we can wrap these in familiar functions and not access them directly
classes also have a proto object covered here
Store.prototype.insert = function(data){
// invoking the setter
// this keyword points to the class(instantiated object)
this.setData = data
}
now we can update operators.js insert
operators.js
// updating insert(letting the store handle everything)
export function insert(row){
this.store.insert(row)
}
Select.js
I had many ideas for select, mostly inspired by other db's, but I settled on a simple and I believe powerful enough API, for now I want select to just do two things select by ID and query the db based on certain filters.
let's start with select by id as it is simple
export default function select(option = "*"){
// checking if option is a number
if(Number(option) !== NaN){
// return prevents select from running code below this if statement()
// think of it as an early break
return this.store.getByid(+option)
// the +option converts option to a number just to make sure it is
}
// query mode code will be here
}
based on the value of option we select to do one of two select by ID or enter what i call a query mode, to select by id all we need to check is if option is a number, if not we enter query mode
Store.js
we need to add the select by id function to the store
...
Store.prototype.getByid = function(id){
const data = this.getData // get the pointer the data(cause it's private we cannot access it directly)
//object(remember the value by reference concept)
if(data[id]){ // checking if id exists
return data[id] // returning the document
}else{
return "noDoc" // for now a str will do
// but an error object is more appropriate(future worry)
}
}
Simple and now we can get a row by id, query mode is a little bit involved, more code and some helpers
Select.js query mode
The core idea is simple really, I thought of the db as a huge hub, a central node of sort, and a query is a small node/channel connected to the center, such that each query node is self contained, meaning it contains it's own state until it is closed.
example
let a = store.select() // returns a query chanel/node
let b = store.select()
// a is not aware of b, vice versa,
//whatever happens in each node the other is not concerned
for this to work we need to track open channels and their state as the querying continues, an object is a simple way to do just that.
const tracker = {
id: 0, // needed to ID each channel and retrieve or update it's state
}
function functionalObj(store){
this.id = NaN // to give to tracker.id(self identity)
}
export default function select(option = "*"){
...
// query mode code will be here
// functionalObj will return the node/channel
return new functionalObj(this.store)
}
functionalObj will have four functions:
beginQuery - will perform the necessary setup to open an independent channel/node to the db
Where - will take a string(boolean operators) to query the db e.g Where('age > 23') return all docs where the age is bigger than 23
endQuery - returns the queried data
close - destroys the channel completely with all it's data
beginQuery
...
function functionalObj(store){
...
// channelName will help with Identifying and dubugging for the developer using our db
this.beginQuery = (channelName = "") => {
// safeguard not to open the same query/channel twice
if(tracker[this.id] && tracker[this.id].beganQ){ // checking if the channel already exists(when this.id !== NaN)
console.warn('please close the previous query');
return
}
// opening a node/channel
this.id = tracker.id
tracker[this.id] = {
filtered: [], // holds filtered data
beganQ: false, // initial status of the channel(began Query)
cName : channelName === "" ? this.id : channelName
}
tracker.id++ // for new channels
// officially opening the channel to be queried
// we will define the getAll func later
// it basically does what it's says
tracker[this.id].filtered = Object.values(store.getAll()) // to be filtered data
tracker[this.id].beganQ = true // opening the channel
console.log('opening channel: ', tracker[this.id].cName) // for debugging
}
// end of begin query function
}
update Store.js and put this getAll func
Store.prototype.getAll = function(){
return this.getData
}
Where, endQuery, close
function functionalObj(store){
this.beginQuery = (channelName = "") => {
...
}
// end of beginQuery
this.Where = (str) => {
// do not allow a query of the channel/node if not opened
if(!tracker[this.id] || tracker[this.id] && !tracker[this.id].beganQ){
console.log('begin query to filter')
return
}
let f = search(str, tracker[this.id].filtered) // we will define search later(will return filtered data and can handle query strings)
// update filtered data for the correct channel
if(f.length > 0){
tracker[this.id].filtered = f
}
}
// end of where
this.endQuery = () => {
if(!tracker[this.id] || tracker[this.id] && !tracker[this.id].beganQ){
console.warn('no query to close')
return
}
// returns data
return {data:tracker[this.id].filtered, channel: tracker[this.id].cName}
};
// end of endQuery
this.close = ()=> {
// if a node/channel exist destroy it
if(tracker[this.id] && !tracker[this.id].closed){
Reflect.deleteProperty(tracker, this.id) // delete
console.log('cleaned up', tracker)
}
}
}
Search
// comm - stands for commnads e.g "age > 23"
const search = function(comm, data){
let split = comm.split(" ") // ['age', '>', 23]
// split[0] property to query
// split[1] operator
// compare against
let filtered = []
// detecting the operator
if(split[1] === "===" || split[1] === "=="){
data.map((obj, i)=> {
// mapSearch maps every operator to a function that can handle it
// and evalute it
// mapSearch returns a boolean saying whether the object fits the query if true we add the object to the filtered
if(mapSearch('eq' , obj[split[0]], split[2])){
// e.g here mapSearch will map each object with a function
// that checks for equality(eq)
filtered.push(obj)
}
})
}else if(split[1] === "<"){
data.map((obj, i)=> {
// less than search
if(mapSearch('ls' , obj[split[0]], split[2])){
filtered.push(obj)
}
})
}else if(split[1] === ">"){
data.map((obj, i)=> {
// greater than search
if(mapSearch('gt' , obj[split[0]], split[2])){
filtered.push(obj)
}
})
}
return filtered // assigned to f in Where function
}
function functionalObj(store){
...
}
mapSearch
// direct can be eq, gt, ls which directs the comparison
// a is the property --- age
// b to compare against --- 23
const mapSearch = function(direct, a, b){
if(direct === "eq"){
// comparers defined func below
return comparers['eq'](a, b) // compare for equality
}else if(direct === "gt"){
return comparers['gt'](a, b) // is a > b
}else if(direct === "ls"){
return comparers['ls'](a, b) // is a < b
}else{
console.log('Not handled')
}
}
const search = function(comm, data){
...
}
...
Comparers
actually does the comparison and returns appropriate booleans to filter the data
// return a boolean (true || false)
const comparers = {
"eq": (a, b) => a === b,
"gt": (a, b) => a > b,
"ls": (a, b) => a < b
}
Select should work now, we can query for data through dedicated channels
test.js
testing everything
import db from './index.js'
let store = new db({timeStamp: true})
store.insert({name: "sk", surname: "mhlungu", age: 23})
store.insert({name: "np", surname: "mhlungu", age: 19})
store.insert({name: "jane", surname: "doe", age: 0})
const c = store.select() // return a new node/channel to be opened
c.beginQuery("THIS IS CHANNEL C") // opening the channel and naming it
c.Where('age < 23') // return all documents where age is smaller than 23
const d = store.select() // return a new node/channel
d.beginQuery("THIS IS CHANNEL D") // open the channel
d.Where('age > 10') // all documents where age > 10
console.log('===============================================')
console.log(d.endQuery(), 'D RESULT age > 10') // return d's data
console.log('===============================================')
console.log(c.endQuery(), "C RESULT age < 23") // return c's data
console.log('===============================================')
c.close() // destroy c
d.close() // destroy d
node test.js
you can actually chain multiple where's on each node, where for now takes a single command
example
const c = store.select()
c.beginQuery("THIS IS CHANNEL C")
c.Where("age > 23")
c.Where("surname === doe") // will further filter the above returned documents
Problems
the equality sign does not work as expected when comparing numbers, caused by the number being a string
// "age === 23"
comm.split(" ") // ['age', '===', '23'] // 23 becomes a string
23 === '23' // returns false
// while 'name === sk' will work
comm.split(" ") // ['name', '===', 'sk']
'sk' === 'sk'
a simple solution will be to check if each command is comparing strings or numbers, which in my opinion is very hideous and not fun to code really, so a solution I came up with is to introduce types for the db, meaning our db will be type safe, and we can infer from those types the type of operation/comparisons
for example a new db will be created like this:
let store = new db({
timeStamp: true,
types: [db.String, db.String, db.Number] // repres columns
})
// if you try to put a number on column 1 an error occurs, because insert expect a string
the next tut will focus on just that.
conclusion
If you want a programming buddy I will be happy to connect on twitter , or you or you know someone who is hiring for a front-end(react or ionic) developer or just a JS developer(modules, scripting etc) I am looking for a job or gig please contact me: mhlungusk@gmail.com, twitter will also do
Thank you for your time, enjoy your day or night. until next time
