File Database in Node Js from scratch part 3: introducing types
Introduction
At the end of this tut or article the aim is simple, having types for each column and a type checking system on insert, for now we will support three types and add more as they are needed.
In this tut, I've experimented a bit with a new type of writing as I am trying to improve my technical writing skills, I hope there is a notable difference, your feedback will be highly appreciated if you have any.
Add types in utils.js
export const types = {
"String": 1,
"Number": 2,
"Object": 3,
}
// you will see how this object is useful in a sec
database.js
now for types to work we have to enforce them early on, meaning a types option when creating a new db is not optional anymore, if it is not present we throw an error
let store = new db("test db", {timeStamp: true,
// mandatory
types: {name:"String", surname:"String", age:"Number"}}
)
I could have used an array to manage types: ["String", "String", "Number"] which would be simpler: an index corresponds with a column, the problem is a document/object {name: "sk", surname: "mhlungu", age: 23} cannot be really trusted to maintain the order of columns, as objects are not by index but keys(for a large enough object the values MAY(i am not sure either and don't want to find out) change positions even if we use object.keys.
so that is why I am mapping column names to their types, which consequently adds a new feature: you cannot add a document with a column that does not exist
e.g {name: "sk", surname: "mhlungu", age: 23} correct
{name: "sk", surname: "mhlungu", age: 23, stack: "React, Ionic"} wrong: must throw an error
let's update database.js to handle this
... // three dots represent previous/existing code
import {types} from "./utils.js"
function db(name,options) {
// if options does not have the types property
if(!options.types){
throw new Error("db needs a types option to work")
}
// checking if the types are supported by the database
const n = Object.keys(options.types).map((val, i)=> {
return types[options.types[val]]
})
...
}
type support check breakdown
const n = Object.keys(options.types).map((val, i)=> {
return types[options.types[val]]
})
// this code is simple really
// for {name:"String", surname:"String", age:"Number"}
// the above loops over the object
// the magic happens in the return, remember our types util:
export const types = {
"String": 1,
"Number": 2,
"Object": 3,
}
// if the type in options.types is present in utils types
// a number corresponding to that type is returned else undefined is returned
// for {name:"String", surname:"String", age:"Number"}
// [1, 1, 2] will be returned
// for {name:"String", surname:"String", age:"Number", stack: "Array"}
// [1, 1, 2, undefined] will be returned
// all we need to do now is check for undefined in the array if we find one
// then we throw an error of unsupported type else we continue and create the db
checking for undefined in database.js
function db(name,options) {
...
if(n.indexOf(undefined) !== -1){ // if we have undefined
const m = Object.keys(options.types)[n.indexOf(undefined)]
// show which column and type is unsupported
throw new Error(`type of ${options.types[m]} for column ${m} does not exist`)
}
// if the above two if's are cleared then we can create the db
this.store = new Store(name, options)
}
Goal one complete we have successfully introduced types, now we need to make sure on insert every document follows the same rules, insert a required type for a column, a column of type string cannot hold a number, that's an error
Store.js - enforcing types on insert
in store's setData we want to end up with something of sort
set setData(data){
// new code
// check if the document has required columns
if(!checkColumns(data, this.#meta.options.types)){
throw new Error(`db expected a document with these columns: ${Object.keys(this.#meta.options.types)},
but got ${Object.keys(data)} for this document ${JSON.stringify(data)}`)
}
// check if the document has correct types
if(!checkTypes(data, this.#meta.options.types)){
throw new Error(`db expected a document with these types: ${Object.values(this.#meta.options.types)},
but got ${Object.values(data)} for this document ${JSON.stringify(data)}`)
}
// new code ends
data._id = this.#meta.length
if(this.#meta.options && this.#meta.options.timeStamp && this.#meta.options.timeStamp){
data.timeStamp = Date.now()
}
this.#data[this.#meta.length] = data
this.#meta.length++
// console.log('data', this.#data)
}
before we write checkColumns and types we need a few utils
in utils.js add :
// return booleans
// () => 👈 functions of sort are called immediate return functions
// they have no {}, they return their values after runnig
export const isStr = (val) => typeof val === "string"
export const isNumb = (val) => typeof val === "number"
export const isObj = (val) => typeof val === "object"
back to Store.js
CheckColumns function
place these func's on top of the class
function checkColumns(doc, types){
let checkOut = true // state -> the most important value here
// if true everything is correct else not
// yes you can definetley use forEach below instead of map(will change it too)
// react.js habits cause me to use map everywhere 😂😂 i just noticed writing the article
Object.keys(types).map((key, i)=> {
if(!checkOut) return checkOut;
if(doc[key] === undefined){
console.log(key, "is missing in this document")
checkOut = false
}
})
if(Object.keys(types).length !== Object.keys(doc).length) checkOut = false
return checkOut
}
explanation:
Object.keys(types).map((key, i)=> {
if(!checkOut) return checkOut; // break out of map early if we already have a
// a column problem
if(doc[key] === undefined){ // if the document has a missing column
console.log(key, "is missing in this document")
checkOut = false
}
})
to notice in the above is that the code will pass even if we have an extra column that does not exist in types Object.keys(types) as we checking columns in types against doc
example:
{name:"String", surname:"String", age:"Number"}
{name: "sk", surname: "mhlungu", age: 23, stack: "React"}
// stack is extra
// the above map will pass cause doc has all types column, the extra will be ignored
// which is wrong, hence the below code to handle this and make sure
// columns are of the same size and we have no extra column
checking for extra columns
if(Object.keys(types).length !== Object.keys(doc).length) checkOut = false
if we found an extra column we return false then insert won't run but throw an error
if(!checkColumns(data, this.#meta.options.types)){
throw new Error(`db expected a document with these columns: ${Object.keys(this.#meta.options.types)},
but got ${Object.keys(data)} for this document ${JSON.stringify(data)}`)
}
if the column check passes then we can check for types
CheckTypes function
import {isStr, isNumb, isObj} from "./utils.js" // typecheck helpers
// basically this function is the same as columns check
function checkTypes(doc, types){
let checkOut = true // state
// map again 🤦♂️, change to forEach please
Object.keys(doc).map((key,i)=> { // looping over the doc keys {name: "sk", surname: "mhlungu", age: 23}
if(!checkOut) return checkOut; // early break
if(types[key] === "String"){ // if the column in question expects a string
if(!isStr(doc[key])) checkOut = false // and the value in doc is not a string throw an error(checkout = false)
}else if(types[key] === "Number"){
if(!isNumb(doc[key])) checkOut = false
}else if(types[key] === "Object"){
if(!isObj(doc[key])) checkOut = false
}
})
return checkOut
}
same thing happens here also if the check types fail insert breaks without inserting, I am one to admit for now error handling is horrible , we cannot just break(which is an assumption the developer is using try catch, which is very rare), I am thinking of a dedicated article to handle errors better maybe returning an object with status, and what happened etc
this will checktypes before running insert code
if(!checkTypes(data, this.#meta.options.types)){
throw new Error(`db expected a document with these types: ${Object.values(this.#meta.options.types)},
but got ${Object.values(data)} for this document ${JSON.stringify(data)}`)
}
what I am noticing so far in these three articles is the vortex abstract API thing we been following is kinda working, look we added a bunch of code, done a lot of refactoring without touching the end point and changing much of previous code, that is indeed a victory 🍾👌🎉, our end point is still clean in index.js no plumbing yet:
import db from "./database.js"
export default db
no shade to plumbers by the way, plumbing or plumber is a football(soccer) slang from my country, meaning a coach who looks promising but is doing absolute rubbish in tactics and formation while having a quality team, which is losing by the way, by plumbing code I mean something similar.
we have basically achieved both goals we set out in the beginning, but remember the main goal was to assist the where function from the previous article with transforming age > 23 string commands to proper values without trying much
let's do that now,
Select.js
Remember our vortex analogy, code that does not concern itself with certain data or state or does not need or require directly must ask the responsible end point for it, so here Select will need types so select must ask Store for them meaning we need a function to return types from the store.
in store.js
// add under set setData inside the class
get getTypes(){
return this.#meta.options.types
}
our proto to get types
Store.prototype.types = function(){
return this.getTypes
}
back to select, because types will be used by the entire channel(possibly in the future), we can add them in the tracker for each channel, this will make it so on channel destruction the types are destroyed also(saving memory)
update beginQuery with code followed by new code comment
this.beginQuery = (channelName = "") => {
// prepare
console.log("creating channel", channelName)
if(tracker[this.id] && tracker[this.id].beganQ){
console.warn('please close the previous query');
return
}
// keys = this.store.allKeys()
this.id = tracker.id
tracker[this.id] = {
filtered: [],
types: {}, // new code
beganQ: false,
cName : channelName === "" ? this.id : channelName
}
tracker.id++
tracker[this.id].filtered = Object.values(store.getAll())
tracker[this.id].types = store.types() // new code
tracker[this.id].beganQ = true
console.log('opening channel: ', tracker[this.id].cName)
// console.log('tracker obj', tracker)
};
update where also to pass the types to search, we can pass the id but it's not that necessary if we can pass the types directly
//update where
// now search takes three arguments
// command, data and types
let f = search(str, tracker[this.id].filtered, tracker[this.id].types)
next we need to update search, for now all we need to know in search is does the command have a number and convert that str number to an actual number, solving our previous problems 23 === "23" // returning false
const search = function(comm, data, types){
let split = comm.split(" ")
// new
if(types[split[0]] === 'Number'){
split[2] = +split[2] // converting "23" to 23 (number)
}
// Problems split[0] is a column
// if the column does not exist cause in where you can pass
// a column that does not exist e.g "aged > 23"
// we need to handle that gracefully
// for now it will fail silently because of the way `where` works
// and return unfiltered data
...
}
that is it for this article you can experiment with test.js, with that we have types finally, and things are getting exciting honestly, I am thinking on moving to dumping the data to a file next. to fulfill the file part in file database, we will deal with other CRUD functions later
