User:Bralani/GSoC2024/Log

Community Bonding Period

• Familiarizing with previous work,especially [Neural Intersection Functions](https://arxiv.org/abs/2306.07191)

Monday, July 1st, 2024

Downloaded the code from the repo of fall rainy in order to have a common base source code. Then, installed different libraries (like pytorch) to make it work.

Tuesday, July 2st, 2024

Implemented a different version of the neural network (NIF) to make it work with hit/miss task. Got a 0.99 accuracy with bounding sphere approach in the training set.

Wednesday, July 3st, 2024

Compared a normal rendering with the neural network (NIF) one: results are still far from being acceptable.

Thursday, July 4st, 2024

Got an average accuracy of 50% with NIF in predicting the true rendering. Tried with a simple KNN with a billion rays stored, got always 50% of accuracy. -> This means that there are some errors in the pipeline (maybe in generating the rays or conversions in spherical coordinates) because the KNN should be better than the random classifier with a billion rays of training set.

Friday, July 5st, 2024

Solved issues with rendering, got a maximum accuracy of 98.6% with the true rendering (bounding sphere sampling). Pay attention that this result has been achieved only with a single render.

Saturday, July 6st, 2024

Implemented a benchmark function that generate different angles of the object (test set). Implemented a different version of loss function (symmetric). Got a maximum accuracy of 98.8% and this result is unbiased -> high confidence.

Sunday, July 7st, 2024

Implemented a grid encoding approach => results are lower than 98.8%.

Monday, July 8st, 2024

Tried with different optimizers, results improved to 99.1% with Nesterov Adam that converges faster and it is very stable between epochs.

Tuesday, July 9st, 2024

Started organizing source code to better understand the workflow.

Wednesday, July 10st, 2024

Continuing organizing source code.

Friday, July 12st, 2024

Finished organizing source code.

Sunday, July 14st, 2024

Added acceleration and support for Metal (Mac OS).

Monday, July 15st, 2024

Implemented a positional encoding like in NeRF's work -> results improved to 99.3% (accuracy).

Tuesday, July 16st, 2024

Implemented an importance sampling approach so as to gather more samples in uncertain areas -> results improved to 99.4% (accuracy) and 99.1% (F1).

Wednesday, July 17st, 2024

The main drawback of current NN is that around boundaries it is not so precise. Time to study more complex encodings: I mention useful papers like DeepSDF, Neural Pull, 2D/3D Gaussian Splatting.

Thursday, July 18st, 2024

Added vector direction as input feature for the NN.

Friday, July 19st, 2024

Tried with SDF neural network, results are not satisfactory because SDF associated to rays are not well defined.

Saturday, July 20st, 2024

Reading more in details papers of Gaussians splatting.

Sunday, July 21st, 2024

Implemented a first version of 3D Gaussians splatting.

Tuesday, July 23st, 2024

Associated a single Gaussian to each positive hit in the training set. The NN must find the largest variance of each Gaussian so as to correctly classify also the negative rays.

Wednesday, July 24st, 2024

Continuing coding Gaussian splatting. Had an issue with memory, understood the problem (broadcasting).

Thursday, July 25st, 2024

Solved the problem with memory, had good results with 10k rays (better than NIF architecture). However, inference is pretty slow and at the moment it is not possible to train with more rays. Moreover, there are still problems with numbers too big or too small that can break the training.

Thursday, July 26st, 2024

Implemented parallelism of Gaussian splatting architecture and inserted double precision floating points.

Saturday, July 28st, 2024

Sped up the inference process by taking only the closest gaussians to the ray to predict. (similar tecnique to the grid encoding)

Monday, July 29st, 2024

Implemented an adaptive learning technique so as to have a variable number of gaussians during the training process (increase the gaussians in the areas more uncertain and decrease the gaussians whose determinant is very small -> small ellipsoid in 3D).

Tuesday, July 30st, 2024

Training with more samples, it is pretty slow.

Wednesday, July 31st, 2024

Predicted images are far from ground truth and moreover the inference is very slow. It cannot substitute the ray tracing pipeline.

Thursday, August 1st, 2024

Studying an interesting paper of SIGGRAPH 2024 (NVBH): https://weiphil.github.io/portfolio/neural_bvh

Friday, August 2st, 2024

Implemented the multi-resolution hash grid.

Saturday, August 3st, 2024

Sampled N points along each ray and validated the multi-resolution hash grid with 3 millions ray. Got convergence in few seconds and metric F1 is about 0.985.

Tuesday, August 6st, 2024

Got F1 metric on 0.998 with 26 million samples and high resolution hash grid. However, inference is slow so there is the need to implement the BVH approach.

Thursday, August 8st, 2024

Implemented a more aggressive sampling part and got 0.9991 as F1 metric.

Monday, August 12st, 2024

Recording rendering times for the nbvh network.

Wednesday, August 14st, 2024

Implemented a 3D visualizer with frames rendered directly by the NN.